RisingWave

Streaming

RisingWave is an open-source distributed streaming database targeting real-time analytics and event-driven applications. It uses the incremental computation model to process streaming data with low latency. RisingWave implements a traditional change propagation framework to keep user-defined materialized views up-to-date. An incremental checkpoint mechanism is used to ensure data consistency. It also has an elastic multi-node architecture with separate data and compute nodes. RisingWave is built from scratch with Rust and is wire compatible with PostgreSQL.

History

RisingWave Database is built by RisingWave Labs (formerly known as Singularity Data), a database systems startup founded in 2021 by former IBM researcher and Amazon Redshift engineer Yingjun Wu.

While working at Amazon Redshift, Wu noticed that existing database systems cannot process streaming data efficiently and existing streaming systems were too complicated for most companies to use. These observations motivated Wu to found RisingWave Labs with a mission to “democratize stream processing”.

Storage Organization

Log-structured

RisingWave uses a LSM-Tree based key-value storage engine that provides MVCC read and write capabilities. All key-value pairs are stored in block-based sorted strings tables (SST). Each SST consists of two files:

  • The .data file is composed of 64KB blocks containing the actual key-value pairs.
  • The .meta file contains metadata such as min-max index, Bloom filter, and per-block metadata.

Checkpoints

Non-Blocking Consistent

RisingWave uses the Chandy–Lamport algorithm to create consistent checkpoints.

To ensure that data is correct and consistent, read queries always fetch data from the most recent checkpoint. This means RisingWave does not ensure read-after-write consistency.

A local shared buffer is used to stage uncommitted write batches submitted by operators. The storage manager will notify all operators to commit their buffered writes into the shared storage when the checkpoint trigger message has reached all operators.

Concurrency Control

Not Supported

RisingWave does not support concurrency control in its compute engine. However, it does employ Multi-version Concurrency Control (MVCC) in its storage engine.

Data Model

Relational

RisingWave uses a relational data model. Its relational tables are composed of a list of strongly-typed columns. All columns are implicitly nullable. The supported primitive data types are boolean, integer, fixed-point and floating-point numbers, strings, and temporals. Composite data types of struct and list are also supported.

Foreign Keys

Not Supported

Indexes

Not Supported

RisingWave does not support traditional index data structures. Instead, indexes are implemented as specialized materialized views. RisingWave stores materialized views as key-value pairs in log-structured merge trees.

Isolation Levels

Not Supported

Joins

Nested Loop Join Hash Join Index Nested Loop Join

RisingWave supports hash join, nested loop join, and index nested loop join (also called lookup join). RisingWave has two execution modes: the batch-query mode and the streaming mode. All three join strategies are used in the batch-query mode, while only hash join and lookup join are used in the streaming mode. The supported join types are inner join, left outer join, right outer join, and full outer join. Window joins are also available as part of RisingWave's support for time window functions.

Query Compilation

Not Supported

Query Execution

Vectorized Model

RisingWave adopts the top-to-bottom vectorized processing model. Operators emit a Data Chunk in batch-query mode and a Stream Chunk in streaming mode. A Data Chunk consists of multiple columns and a visibility array (represents each row's visibility status for row filtering purposes). A Stream Chunk consists of multiple columns, a visibility array, and an additional ops column marking an operation on each row. Each entry in the ops column can be one of Delete, Insert, UpdateDelete, or UpdateInsert.

Query Interface

SQL

The RisingWave SQL query interface is mostly compatible with PostgreSQL. It has client libraries in Java, Node.js, Python, and Go. RisingWave is wire compatible with PostgreSQL and can be accessed with PostgreSQL terminal psql. As a cloud-native database, RisingWave can be integrated with cloud services such as Confluent Cloud, DataStax, and Grafana Cloud. To support stream processing, RisingWave can be integrated with the following message brokers or streaming services: Kafka, Redpanda, Apache Pulsar, DataStax Astra Streaming, StreamNative Cloud, and Kinesis Data Streams.

Storage Architecture

Disk-oriented

RisingWave has a disk-oriented storage architecture. Files are directly written to an S3-compatible shared storage service by default. RisingWave also supports local drives, Google Cloud Storage, and HDFS/WebHDFS as shared storage destinations.

Stored Procedures

Supported

RisingWave offers UDF implemented as external functions in Python.

System Architecture

Shared-Everything

RisingWave adopts a disaggregated architecture to support elastically scaling the compute nodes without migrating the storage. A RisingWave database is composed of four key layers:

  • A serving layer that parses, plans, and optimizes SQL queries.
  • A processing layer that performs all the data computations and updates.
  • A persistence layer that stores and retrieves data on an object storage service (such as S3).
  • A meta server that coordinates the operations between the serving layer and the processing layer.

The serving, processing, and persistent layers each consist of multiple nodes. Compute nodes in the processing layer execute the optimized query plan (generated by the serving layer) in parallel. Compactor nodes in the persistent layer compact data with consistent hashing and upload the resulting SST files to shared storage.

The meta server is responsible for many tasks, including but not limited to:

  • Managing the stream graph of RisingWave
  • Managing barrier injection and collection for initiating checkpoint capture
  • Manages the cluster information, such as the address and status of nodes
  • Detecting failure by periodically sending heartbeats to serving-layer and processing-layer nodes

Views

Virtual Views Materialized Views

RisingWave supports both views and materialized views. New materialized views can be created based on existing materialized views. RisingWave updates materialized views incrementally as new data is fed into the database.

RisingWave Logo
Website

https://www.risingwave-labs.com/

Source Code

https://github.com/risingwavelabs/risingwave

Tech Docs

https://www.risingwave.dev/docs/latest/intro/

Twitter

@RisingWaveLabs

Developer

RisingWave Labs

Country of Origin

US

Start Year

2021

Project Type

Commercial, Open Source

Written in

Rust

Supported languages

Go, Java, JavaScript, Python

Compatible With

PostgreSQL

Operating Systems

Linux

Licenses

Apache v2