RisingWave

View Current Viewing Revision #14 from 05/01/2023 8:10 p.m.

Streaming

RisingWave is an open-source distributed streaming database targeting real-time analytics and event-driven applications. It uses the incremental computation model to process streaming data with low latency. RisingWave implements a traditional change propagation framework to keep user-defined materialized views up-to-date. An incremental checkpoint mechanism is used to ensure data consistency. It also has an elastic multi-node architecture with separate data and compute nodes. RisingWave is built from scratch with Rust, and is wire compatible with PostgreSQL.

History

RisingWave Database is built by RisingWave Labs (formerly known as Singularity Data), a database systems startup founded in 2021 by former IBM researcher and Amazon Redshift engineer Yingjun Wu.

While working at Amazon Redshift, Wu noticed that existing database systems cannot process streaming data efficiently and existing streaming systems were too complicated for most companies to use. This observation motivated Wu to found RisingWave Labs with a mission to “democratize stream processing”.

Checkpoints

Non-Blocking Consistent

RisingWave uses the Chandy–Lamport algorithm to create consistent checkpoints.

To ensure that data is correct and consistent, read queries always fetch data from the most recent checkpoint. This means RisingWave does not ensure read-after-write consistency.

A local shared buffer is used to stage uncommitted write batches submitted by operators. The storage manager will notify all operators to commit their buffered writes into the shared storage when checkpoint trigger message has reached all operators.

Compression

Naïve (Page-Level) Prefix Compression

RisingWave applies both naive compression and prefix compression at block-level. It uses LZ4 and Zstd for naive compression.

Concurrency Control

Not Supported

RisingWave does not support concurrency control in its compute engine. However, it does employ Multi-version Concurrency Control (MVCC) in its cloud-native LSM-Tree-based storage engine.

Data Model

Relational

RisingWave uses a relational data model. Relational tables are composed of a list of strongly-typed columns. All columns are implicitly nullable. RisingWave supports primitive data types of: boolean, integer, fixed-point and floating-point numbers, strings, and temporals. Composite data types of struct and list are also supported.

Foreign Keys

Not Supported

Indexes

Not Supported

RisingWave does not support traditional index data structures. Instead, index is implemented as a specialized materialized view. RisingWave's storage engine stores materialized view as key-value pairs in log-structured merge trees.

Isolation Levels

Not Supported

Joins

Nested Loop Join Hash Join Index Nested Loop Join

RisingWave supports join with hash join, nested loop join, and index nested loop join (also called lookup join). RisingWave has two execution modes: the batch-query mode and the streaming mode. All three join strategies are used in the batch-query mode, while only hash join and lookup join are used in the streaming mode. The supported join types are: inner join, left outer join, right outer join, and full outer join. RisingWave supports time window functions, window joins are also supported.

Logging

Not Supported

Parallel Execution

Bushy

RisingWave supports bushy parallelism in both its batch-query and streaming modes. Consistent hashing is employed to partition data for parallel execution.

Query Compilation

Not Supported

Query Execution

Vectorized Model

RisingWave adopts a vectorized model.

Query Interface

SQL

The RisingWave SQL query interface is mostly compatible with PostgreSQL. It has client library in Java, Node.js, Python, and Go. RisingWave also offers UDF API for Python. RisingWave is wire compatible with PostgreSQL, and can be accessed with PostgreSQL terminal psql. As a cloud-native database, RisingWave can be integrated with cloud services such as Confluent Cloud, DataStax, and Grafana Cloud. To support stream processing, RisingWave can be integrated with the following message brokers or streaming services: Kafka, Redpanda, Apache Pulsar, DataStax Astra Streaming, StreamNative Cloud, and Kinesis Data Streams.