History[01]

Developed by Uber to meet their specific need "to make similar queries over relatively small, yet highly valuable, subsets of data (with maximum data freshness) at high QPS and low latency," with queries such as time series aggregations over geofences.

Checkpoints[04]

Snapshots are triggered by either a certain number of mutations or a certain time frame specific to each table.

Compression[01]

AresDB only compresses data with user defined sort orders that have low cardinality.

Data Model

Foreign Keys[05]

AresDB supports foreign key joins.

Hardware Acceleration[01]

AresDB uses GPUs for its query execution.

Indexes[01]

AresDB uses Hash Tables primarily for primary key deduplication.

Joins[05][01]

AresDB supports hash joins from fact tables (finite set data such as cities) to dimension tables (infinite streaming data such as rides). The database also supports geospatial joins (i.e. geographically bounded area overlap) and normal foreign key joins. Note that AresDB uses late materialization for its joins, meaning the join may not be executed until a foreign key is accessed.

Logging[06]

Log files contain description of database upserts which must be replayed to rebuild the database after a crash.

Parallel Execution[07]

Executes queries with the one operation per kernel (OOPK) model.

Query Compilation

Query Execution

AresDB works with vector batches that are efficiently processed in parallel using the Thrust library.

Query Interface[05]

AresDB uses a proprietary execution language called Ares Query Language (AQL) which is based in the JSON format, making it compatible with any language that can handle files and/or JSON.

Storage Architecture

Data within the archival delay of a table is kept uncompressed in live batches, while everything else is stored in compressed archival batches. If new data is ingested that is outside the archival array, it's added to an archival backfill queue which will be inserted into the archived batches asynchronously.

Storage Model

Storage Organization[08][09]

Archived data is sorted in a user specified column order, and files are organized by UTC day and Unix time cutoffs.

Stored Procedures

System Architecture[01]

The CPU is only used to load information from storage into CPU memory and to distribute this data to GPU memory. The database system delegates each operator in a query to some GPU, so it's able to handle multiple GPUs by delegating different operations to different GPUs, each of which have completely separate memory. There are plans to implement proper distributed designs, but currently we're limited to a single system with multiple GPUs.

Views[05]

AresDB uses late materialization for its joins, meaning that it may only physically execute the join once a foreign key is accessed.