History[05]

OpenTSDB was originally written by Benoit Sigoure in 2010 to monitor metrics of the StumbleUpon search engine which requires storing over 1 billion data points per day. StumbleUpon was in charge of the initial development and its open-source release. Yahoo! is currently maintaining OpenTSDB along with the open-source community.

Compression[06]

OpenTSDB uses Row Compaction to compress data. Whenever a cell is to be written, its row key is pushed into a compaction queue. There is a separate thread that periodically goes through the queue and aggregate data with the same key into a big cell. It then writes the big cell and deletes the individual cells in the queue. This process is effective because in HBase the row key is repeated for every single cell, and there is no way to efficiently append byte at the end of a cell.

At data level, OpenTSDB uses LZO compression algorithm.

With these two techniques, OpenTSDB is able to reduce the average size of one data point from 12 bytes to 2-3 bytes.

Concurrency Control[07]

OpenTSDB avoids multiple writers creating duplicate rows in the case of writer restart by making writes idempotent. It enforces a fixed timestamp boundary for each row. When a write reconnects to HBase, it will always write to the appropriate row according to the timestamp instead of creating new rows.

Data Model

Foreign Keys

Joins

Parallel Execution

Query Compilation

Query Interface

Storage Architecture

Storage Model

Storage Organization

Stored Procedures

System Architecture