History

HStreamDB is built by EMQ, a company providing open source IoT data infrastructure. It was first open sourced in 2021 and is under active development by the Haskell Team from EMQ.

HStreamDB was developed to incorporate a data-driven model to efficiently process stream data in a database. In contrast to the command-driven model of most databases which analyzes data when the client sends a request, HStreamDB’s goal was to analyze data as it is ingested in real-time and deliver the analyses with low-latency.

Compression[04]

Compression is used to reduce network bandwidth utilization when transferring data to and from the database. Compression and decompression is performed entirely by the client and the compressed data is stored natively in the database. HStreamDB supports both gzip and zstd compression algorithms.

Concurrency Control[04]

Since records can only be appended to streams, and records can be written out of order, concurrency control is not necessary for HStreamDB.

Data Model[04][05]

HStreamDB models data as records which are written to streams. All records have a unique identifier, and the data in a record can either be an HRecord or a Raw Record. A HRecord can be thought of as a traditional tuple in a database with support for nested maps and arrays. HRecords can be queried using SQL. A Raw Record contains arbitrary binary data which the database does not interpret or query. Raw Records are intended to be consumed from subscriptions.

Foreign Keys

Indexes

Isolation Levels

Joins[06]

HStreamDB supports nested loop joins between two streams and two materialized views. Joins between a stream and materialized view are also supported.

Query Compilation

Query Execution

Query Interface[07][08]

HStreamDB supports interfacing with the database with either SQL or its custom API. It uses a SQL dialect that is a subset SQL-92 with extensions to support stream operations. Queries can be executed from a command line interface and the Java, Go, and Python clients. HStreamDB’s custom API is implemented in its clients and can be used to insert and consume data.

Since HStreamDB is a streaming database, it handles queries differently from a typical database. Queries are treated as running tasks that fetch data from streams and produce results continuously as the streams are updated. HStreamDB also supports subscriptions, where multiple consumers can read data in real-time from a single stream as records are added by producers.

Storage Architecture[09]

HStreamDB is a disk-oriented database that uses the RocksDB storage engine. This allows HStreamDB to support large scale data streams.

Storage Model[09]

HStreamDB does not implement its own storage layer, and instead relies on RocksDB as a key-value store. All of its data is eventually processed and stored in the key-value file format implemented in RocksDB.

Stored Procedures

System Architecture[09][10][11]

HStreamDB is based on a shared nothing architecture. Each of the nodes in a deployment are identical and contain two components, the HStream Server and the HStream Storage layer.

The HStream Server is responsible for parsing and executing SQL queries as well as any other computations required by a connected client.

The HStream Storage layer is a distributed storage system that uses RockDB as a key-value store for all of the streaming data. Streams are stored across multiple nodes in shards, and shards can be replicated using Paxos. This layer is also responsible for pushing data to consumers with stream subscriptions.

Views[12]

HStreamDB supports incrementally updated materialized views. As data is added to streams, views are updated in real-time. This makes querying views fast since they always contain the latest result. Views are different from streams since they are only stored in memory.