Storage Organization

Log-structured

The storage is a blend of a Log-Structured Merge-Tree (from RocksDB), inverted indexes, and a columnar data store (from IResearch).

Checkpoints

Consistent

SereneDB ensures data persistence through consistent checkpoints and a write-ahead log (WAL). Every now and then it takes a full snapshot of the data and saves it to disk. In the intervals between these snapshots, every change is recorded in the WAL. This system allows the database to automatically recover after a crash: it loads the last saved checkpoint and then uses the WAL to re-apply all subsequent changes. For immediate data persistence, write operations can be configured to wait until the WAL journal has been fully written to the disk.

Compression

Dictionary Encoding Run-Length Encoding Naïve (Page-Level) Bit Packing / Mostly Encoding

Concurrency Control

Multi-version Concurrency Control (MVCC)

To handle transactions and real-time updates, SereneDB utilizes Multi-Version Concurrency Control. This is inherited from its use of RocksDB as a core component of the storage engine.

Data Model

Relational

The system uses a relational data model and aims for high compatibility with PostgreSQL.

Indexes

Inverted Index (Full Text) Log-Structured Merge Tree

The default index structures are the LSM-tree provided by RocksDB for primary data organization and inverted indexes from IResearch for full-text search. SereneDB also supports vector, geospatial for specialized search and analytics use cases.

Isolation Levels

Snapshot Isolation

The system supports Snapshot Isolation, which is the default level.=

Joins

Hash Join Sort-Merge Join Index Nested Loop Join

SereneDB leverages the join algorithms provided by the Velox execution engine. SereneDB supports search index bitset join.

Logging

Physical Logging

Query Execution

Vectorized Model

The system uses the Velox query execution engine, which employs a vectorized execution model. Operations are processed in batches of values (vectors) on a column-by-column basis.

Query Interface

SQL

SereneDB uses SQL as its primary query interface, specifically supporting the PostgreSQL dialect. Its support for the PostgreSQL wire protocol allows it to be used with standard PostgreSQL clients and tools.

Storage Architecture

Disk-oriented

SereneDB is a disk-oriented database that supports larger-than-memory datasets. It utilizes a Hybrid Storage Engine that combines disk-based components (RocksDB, search index segment files) with in-memory data structures.

Storage Model

Decomposition Storage Model (Columnar) Hybrid

RocksDB-based layout for incremental updates. Each table row of N columns will be split into a set of N key-value pairs stored in sorted column order. Search columnar layout for persistent storage. Columns are split into a set of compressed data blocks for aggregations.

System Architecture

Shared-Nothing

SereneDB systems consist of shard groups: groups of identical instances hosted on different underlying servers, serving the same data and replicating continuously to ensure high availability as well as providing additional query capacity. Larger data sets can be range or hash partitioned over multiple shard groups. In a sharded cluster, the values of one or more fields known as the shard key determine which shard group hosts that specific record.

Views

Virtual Views

SereneDB supports standard SQL views and Search views to union query to multiple search indexes.