Indexes

BitMap Inverted Index (Full Text)

Pinot supports pluggable indexing technologies like Sorted Index, BitMap Index, and Inverted Index. BitMap Index is used to optimize queries on categorical data. Inverted Index is used to support lookup by key word. They are chosen to leverage the features of social data: usually categorical and textual.

Inverted Index can be built based on BitMap. And BitMap Index can be optimized with various compression techniques. It can also be physically reordered to optimize some specific queries in Pinot, since filters on such column usually target a contiguous range of the column data.

Query Execution

Tuple-at-a-Time Model Vectorized Model

Storage Organization

Heaps

Pinot stores segments in directories of UNIX filesystem. Each such directory contains a metadata file and an index file. The metadata file stores information about record columns in the segment. The index file stores indexes for all the columns. The global metadata about segments, including the mapping of a segment to its position, is maintained in controller clusters.

Concurrency Control

Not Supported

Pinot moves the execution of queries to segments. There will be no race condition in the server-side query execution since segments are immutable.

System Architecture

Shared-Nothing

Pinot consists of four parts: servers, controllers, brokers, and minions. They together support the functionality of data storage, data management, and query processing.

Servers

Servers are responsible for data storage. Pinot stores segments in each server node in a distributed manner. Each segment has multiple replicas and transactions are executed in active-active manner.

Controllers

Controllers are responsible for maintaining global metadata. They are implemented with Apache Helix and Zookeeper.

Brokers

Brokers are responsible for query routing. They control the flow of query such as where each query should go to and how to generate the final result with intermediate results from different nodes.

Minions

Minions are responsible for running maintenance tasks, which are usually time consuming and should not influence the running queries.

Data Model

Relational

Pinot is a relational datastore. The data type of each attribute can be integers with various length, floating-point numbers, strings, booleans, and arrays. The column type of each attribute can be dimensions, metrics, and time.

Query Interface

Custom API

Pinot uses Pinot Query Language (PQL) as its query interface. It is a subset of SQL. PQL supports queries like selection, projection, aggregations, and top-n. But it does not support joins, nested queries, record-level creation, updates, deletion, or any data definition language (DDL).

Joins

Not Supported

The query interface of Pinot, Pinot Query Language (PQL), does not support joins.

Checkpoints

Not Supported

Pinot uses replicas to provide fault tolerance and high availability. It also uses redundant controller instances to improve availability.

However, checkpoints are not supported since segments are immutable, which means there will be no write on segments during the execution of queries. But it's possible for a segment to be entirely replaced with a newer version.

Storage Model

Hybrid

Pinot uses a hybrid data model, which divides rows into segments and stores data inside each segment in Columnar manner. A segment is a basic unit of replication. It's immutable and typically contains tens of millions of rows.

Compression

Dictionary Encoding Run-Length Encoding Bitmap Encoding Bit Packing / Mostly Encoding

Pinot leverages various types of encoding to reduce storage overhead. The typical size of a segment varies from a few hundred megabytes up to a few gigabytes. Different data encoding techniques have different specialized physical operators to optimize query execution.