Apache Druid is an open-source distributed real-time analytics database designed for business intelligence (OLAP) queries on streaming and historical data. It is optimized for time series scans and aggregations. It supports loading data from both deep storage system like HDFS and streaming sources like Kafka. Internally, Druid uses Zookeeper for cluster node coordination, a relational database like MySQL or Postgres to keep track of metadata, and a deep storage system such as HDFS for storing data. Druid also has low latency between the event creation and when it can be queried, which makes Druid desirable for real-time analytics. Druid stores incoming data in a unique format called a segment to allow fast aggregations for arbitrary dimensionalities of data. Druid is commonly used to power GUI-based analytical BI apps via JDBC and as a backend for AI apps via REST API. Druid is also used for clickstream analytics, network telemetry analytics, application performance analytics, advertising analytics.[05][06]
- Website
- https://druid.apache.org[01]
- Source Code
- https://github.com/apache/druid[02]
- @druidio
- Developer
- Governance
- Apache Software Foundation
- Country of Origin
- US
- Start Year
- 2011 [07]
- Project Type
- Open Source
- Written in
- Java
- License
- Apache v2
Apache Druid is an open-source distributed real-time analytics database designed for business intelligence (OLAP) queries on streaming and historical data. It is optimized for time series scans and aggregations. It supports loading data from both deep storage system like HDFS and streaming sources like Kafka. Internally, Druid uses Zookeeper for cluster node coordination, a relational database like MySQL or Postgres to keep track of metadata, and a deep storage system such as HDFS for storing data. Druid also has low latency between the event creation and when it can be queried, which makes Druid desirable for real-time analytics. Druid stores incoming data in a unique format called a segment to allow fast aggregations for arbitrary dimensionalities of data. Druid is commonly used to power GUI-based analytical BI apps via JDBC and as a backend for AI apps via REST API. Druid is also used for clickstream analytics, network telemetry analytics, application performance analytics, advertising analytics.[05][06]
History[07][08]
Druid was originally developed by engineers at Metamarkets to solve the problem of analyzing high dimensional data set in real-time. Scan and aggregation of billions of records in traditional relational databases are not fast enough, and pre-computing aggregations with NoSQL architecture requires unacceptably long processing time which creates high latency between event occurrence and its availability for querying. Druid was released in April, 2011 to address the need for fast, real-time analytics for high dimensional time series data. It was open sourced in Oct, 2012 and is under active development.
Query Interface[13]
Druid uses customized query interface expressed in JSON for metadata, aggregation and search. Druid provides support for SQL via Apache Calcite.
Storage Architecture[14][15]
Druid was built with all in-memory. However such choice is costly given large amount of data. It then switches to use a combination of memory and disk pages and allow users to customize the behavior.
Storage Model[16]
Druid uses segments files to stores its index. A segment file is a basically a columnar storage model consists of three basic column types: timestamp columns, dimension columns and metric columns. This structure allows fast aggregation across different fields.
Citations
17 sources- Apache Druid | Apache® Druid apache.org
- GitHub - apache/druid: Apache Druid: a high performance real-time analytics database. · GitHub github.com
- Introduction to Apache Druid | Apache® Druid apache.org
- Apache Druid - Wikipedia wikipedia.org
- Technology | Apache® Druid apache.org
- Powered by Apache Druid | Apache® Druid apache.org
- https://druid.apache.org/blog/2011/04/30/introducing-druid.html apache.org
- http://static.druid.io/docs/druid.pdf druid.io
- https://druid.apache.org/blog/2012/10/24/introducing-druid.html apache.org
- Joins | Apache® Druid apache.org
- https://druid.apache.org/docs/latest/dependencies/deep-storage.html apache.org
- Query execution | Apache® Druid apache.org
- Druid SQL overview | Apache® Druid apache.org
- Frequently Asked Questions | Apache® Druid apache.org
- Introduction to Apache Druid | Apache® Druid apache.org
- Segments | Apache® Druid apache.org
- https://github.com/apache/druid/commit/3d1e0d03de341b41027dfe3d493800274f5e895c github.com