LinDB is a distributed time-series DBMS written in Go. It is optimized especially for real-time writing and retrieval tasks, and is used internally by ELEME Inc. The internal version is Java based, however open sourced version is redesigned in Go. It requires Zookeepers to store cluster metadata.
Companies like ELEME Inc. need to monitor all its systems, which calls for DBMS capable of saving multi dimension monitoring metrics for several PB per day. Previously the company use graphite, however its performance downgrades when the dimension of the metrics increases. LinDB is developed to address this demand. The project is launched at 2016 internally, and has been updated three versions. The open sourced version is a redesign and rewrite in Go since 2019, however is not in production stage yet.
Similar to other time-series DBMS, the data model contains four main components: timestamp, metrics, tags, and fields. The metric is the name of the measurement attribute. Tags are combinations of dimensions of measurement used during query, and the fields stores metric data. User has to define whether their columns belong to tags or fields.
LinDB uses a combination of bitmap encoding and data compression. Since it is designed for time-series data, all data entry contains a timestamp and several multi dimensional metrics, such as ip string, shard id, etc. The timestamp is saved with a base timestamp and a bitmap with each bit encoding a time within some time period following the base timestamp. The data blocks are compressed with xor, in reference to Gorilla.
LinDB is designed to separate the computation and storage in a distributed scenario, in order to support both fast insert and high throughput read, including mainly aggregation operations such as group by.
The storage layer LinStorage mainly stores data with write replication, with shard number configurable to user. Each cluster contains several servers, and data are redundantly stored across servers.
The computation layer interacts with user via LinProxy. The LinProxy interacts with both users and servers. Users submit queries to the proxy, then the query is parsed and redirected to server. Results are returned and aggregated inside the LinProxy.
https://github.com/eleme/lindb
https://github.com/eleme/lindb
https://github.com/eleme/lindb/wiki
ELEME Inc.
2019