History[04][06]

Meta's data infrastructure contains dozens of specialized data computation engines, which have been largely developed independently. Maintaining and enhancing each of them can be difficult, especially considering the rapid change of workload requirements and hardware condition.

Velox is created in 2020 and open-sourced in 2021 to address this problem as a unified execution engine. It is under active development, but it is already in various stages of integration with some systems, including Presto, Spark, and PyTorch (the latter through a data preprocessing library called TorchArrow), etc. Additional contributions were provided by Intel, ByteDance, and Ahana.

Checkpoints[04][07]

Does not support database checkpoint. However, there is some action to make the executor more resilient. When memory allocation fails in a task, the current state can be spilled to disk and the task can be paused and return a continuable VeloxPromise.

Compression[04][05]

Vectors in Velox is arrow compatible with slight difference. During execution, the vector being passed around may be compressed based on its value feature. There are flat vector, dictionary vector, lazy vector or constant vector, etc. Vectors typically contains four lists: Null Value Mask, Offsets, Sizes, Elements.

For strings, it adopts the Umbra way of store 4-byte string prefix plus 8-byte pointer to BLOB buffer.

Concurrency Control[04]

Velox does not deal with concurrency control.

Data Model[04][08]

Velox uses Relational model. Inside the Velox dataframe abstraction, there can be scalar values including numbers of different length and precision, strings (VARCHAR or VARBINARY), timestamps of different precision and pointer to functions. Complex types in Velox include arrays, maps, and structs.

Datatable can contain integers and floats of different precision, strings (both varchar and varbinary types), dates, timestamps, and functions (lambdas). It also supports complex types such as arrays, fixed-size arrays (used to implement ML tensors), maps, and structs.

Velox also provides a data type called OPAQUE that can wrap arbitrary C++ data structures.

Foreign Keys

Indexes[04]

Velox does not support index. But it supports expression pushdown to linked storage layer.

Isolation Levels

Logging

Parallel Execution[04][09][10]

The top level concept in Velox execution is the query plan, a.k.a Task. Task can then be converted to multiple stacked pipelines, which is similar to idea of pipeline in Morsel-Driven Parallelism. Each Pipeline is consist of multiple Nodes, such as HashJoinNode, CrossJoinNode, MergeJoinNode, LocalMergeNode, LocalPartitionNode and TableScanNode. Nodes complete their execution using Operators and Drivers, which are both created by the Task. Each Driver is a thread and it takes over the ownership of the Operator from Task. One node can have multiple Drivers working at the same time to achieve inner-query parallelism.

Query Compilation[04]

Right now, Velox experimentally supports query compilation through Codegen. It will transpile the query plan (Task) into C++ code and compile it to shared library using regular compilers (gcc / clang). This shared library can be linked to the main process at runtime.

Query Execution[04][11]

Velox adopts a push-based vectorized query execution paradigm.

Query Interface[04]

Users should use C++ for native support. Velox is also built as binary wheels for PyVelox (the Velox Python Bindings).

Storage Architecture[04]

Velox itself does not manage disk storage. It is expected to be connected with a disaggregated storage system such as S3, HDFS and Tectonic.

Storage Format

Storage Model[04]

As an execution engine designed for OLAP systems, DBMS is designed mostly for DSM storage model.

Storage Organization[04]

Velox has a cache system between execution and underlying disaggregated storage to increase cache hit rate. Cache can is in RAM when first use and can be spilled out to disk for later. Velox also supports column prefetching by tracking the access frequency of each column in a query.

Stored Procedures[04][12][13][14]

Velox exposes C++ scalar UDF function API to the user. Users can write business logic based on a template, and Velox will help compile arithmetic functions to SIMD automatically. It also supports user-defined aggregation functions.

System Architecture[04]

Velox focuses on computation efficiency on single computer. Velox itself is an embedded engine. However, depending on the host system, it can also be expanded to run as standalone program. Prestissimo is the example of such practice.

Views[15]

There is materialized view between pipelines (thread-level) stored in local exchange queues. Also, there is materialized view between tasks (computer-level) maintaining by exchange client.