Pinot

Pinot is a distributed relational OLAP datastore written by LinkedIn. It's designed to support large-scale real-time analytics on any given data set. For use cases that are sensitive to data freshness, Pinot is able to directly ingest streaming data from Kafka. For applications that can tolerate a lag time of few hours to a day of data, Pinot is able to ingest batch data from Hadoop. It's also able to dynamically merge data streams that come from both offline and online systems. Pinot uses a hybrid data model. It divides tables to segments, which are sets of tuples. Tuples inside each segment are organized in columnar manner. A segment is a basic unit in Pinot: Data from Kafka or Hadoop will be processed and cached locally as segments in Pinot server nodes; It stores metadata and necessary zone maps for the tuples inside it; Storage optimizations are applied for tuples in a segment; Indexes are built for each segment; Query plans and optimizations are also generated and performed on a per-segment basis. The external building blocks of Pinot are Zookeeper and Apache Helix.

History

Pinot was first developed by LinkedIn in 2014 as an internal analytics infrastructure. It originated from the demands to scale out OLAP systems to support low-latency real-time queries on huge volume data. It was later open-sourced in 2015 and entered Apache Incubator in 2018. Pinot was named after the Pinot noir, name of a grape varietal that can produce the most complex wine but is the toughest to grow and process. It's a portrayal of data: powerful but hard to analyze.

Query Interface

Custom API

Pinot uses PQL query interface, which is a subset of SQL. PQL supports selection, projection, aggregations, and top-n. But it does not support joins, nested queries, record-level creation, updates, deletion or any data definition language (DDL).

Storage Model

Hybrid

Pinot uses a hybrid data model, which divides rows into segments and stores data inside each segment in Columnar manner. A segment is a basic unit of replication. It's immutable and typically contains tens of millions of rows.

Checkpoints

Not Supported

Pinot uses replicas to provide fault tolerance and high availability. It also uses redundant controller instances to improve availability. However, checkpoints are not supported since segments are immutable, which means there will be no write on segments during the execution of queries. But it's possible for a segment to be entirely replaced with a newer version.

Data Model

Relational

Pinot uses relational data model. In terms of data types, attributes in a relation can be integers with various length, floating-point numbers, strings, booleans, arrays, and timestamps. In terms of analyst, attributes can be dimensions and metrics.

System Architecture

Shared-Nothing

Pinot consists of four parts: servers, controllers, brokers, and minions. They together support the functionality of data storage, data management, and query processing. ### Servers Servers are responsible for data storage. Pinot stores segments in each server node in a distributed manner. Each segment has multiple replicas and transactions are executed in active-active manner. ### Controllers Controllers are responsible for maintaining global metadata. They are implemented with Apache Helix and Zookeeper. ### Brokers Brokers are responsible for query routing. They control the flow of query such as where each query should go to and how to generate the final result with intermediate results from different nodes. ###Minions Minions are responsible for running maintenance tasks, which are usually time consuming and should not influence the running queries.

Compression

Dictionary Encoding Run-Length Encoding Bitmap Encoding Bit Packing / Mostly Encoding

Pinot leverages dictionary encoding and bit packing for columns in segments to reduce storage overhead. The typical space a segment consumes varies from hundreds of megabytes to several gigabytes.

Storage Organization

Heaps

Pinot stores segments in directories of UNIX filesystem. Each such directory contains a metadata file and an index file. The metadata file stores information about record columns in the segment. The index file stores indexes for all the columns. The global metadata about segments, including the mapping of a segment to its position, is maintained in controller clusters.

Website

https://engineering.linkedin.com/teams/data/projects/pinot

Source Code

https://github.com/apache/incubator-pinot

Tech Docs

https://github.com/apache/incubator-pinot/wiki

Developer

LinkedIn

Country of Origin

US

Start Year

2014

Project Type

Open Source

Written in

Java

Supported languages

Java

Operating Systems

All OS with Java VM

Licenses

Apache v2