History

Companies like ELEME Inc. need to monitor all its systems, which calls for DBMS capable of saving multi dimension monitoring metrics for several PB per day. Previously the company use graphite, however its performance downgrades when the dimension of the metrics increases. LinDB is developed to address this demand. The project is launched at 2016 internally, and has been updated three versions. The open sourced version is a redesign and rewrite in Go since 2019, however is not in production stage yet.

Compression[03][04]

LinDB uses a combination of bitmap encoding and data compression. Since it is designed for time-series data, all data entry contains a timestamp and several multi dimensional metrics, such as ip string, shard id, etc. The timestamp is saved with a base timestamp and a bitmap with each bit encoding a time within some time period following the base timestamp. The data blocks are compressed with xor, in reference to Gorilla.

Concurrency Control[04]

Concurrency control is not supported. All writing to the DB is first directed to the specific shard based on its tagKey and tagValue. Within each shard all writing is handled by a single thread.

Data Model

Query Interface

System Architecture