Apache Kudu is an open source storage engine for structured data that is part of the Apache Hadoop ecosystem. The primary intention of Kudu is to allow applications to perform fast big data analytics on rapidly changing data. It was designed for fast performance for OLAP queries. Built for distributed workloads, Apache Kudu allows for various partitioning of data across multiple servers. Also, being a part of the Hadoop ecosystem, Kudu is supports the use of Apache data processing frameworks like Spark, Impala or MapReduce on its tables.[05][06]
- Website
- https://kudu.apache.org[01]
- Source Code
- https://github.com/apache/kudu[02]
- Tech Docs
- https://kudu.apache.org/docs[03]
- Developer
- Country of Origin
- US
- Start Year
- 2016 [04]
- Project Type
- Open Source
- Written in
- C++
- Derived From
- HBase
- Inspired By
- Calvin
- Operating System
- Linux
- License
- Apache v2
Apache Kudu is an open source storage engine for structured data that is part of the Apache Hadoop ecosystem. The primary intention of Kudu is to allow applications to perform fast big data analytics on rapidly changing data. It was designed for fast performance for OLAP queries. Built for distributed workloads, Apache Kudu allows for various partitioning of data across multiple servers. Also, being a part of the Hadoop ecosystem, Kudu is supports the use of Apache data processing frameworks like Spark, Impala or MapReduce on its tables.[05][06]
History[07]
Prior to Kudu, most data storage engines were able to store one type of structured data, static or mutable. Storage engines for static data were unable to make changes to individual records while storage engines for mutable data had a low throughput for sequential reads. Because of this developers typically used two different storage engines for first mutating their data and then performing analytics. Apache Kudu was designed to support both data formats and provide both high throughput sequential-access and random-access queries. Kudu was developed as internal project at Cloudera and become open to the public in September 2016.
Compression[08]
Each column in a Kudu table can be encoded in certain ways based on the type of that column. By default, bit packing is used for various int, double and float column types, run-length encoding is used for bool column types and dictionary-encoding for string or binary column types. By default Kudu doesn't compress columns but it supports per-column compression using LZ4, Snappy or zlib compression codecs.
Concurrency Control[09][10]
Kudu employs MVCC. Kudu uses an optimistic concurrency model in which readers don't block writers and writes don't block readers. As a result less lock acquisitions are needed during large table scans.
Data Model[08]
Kudu is a relational database. Unlike traditional relation databases, Kudu also utilizes partitioning data into tablets that are stored on individual servers. All rows within a tablet are ordered by a primary key.
Indexes[12][11]
Currently Kudu does not support any additional indexes (aside from the primary index). As an alternative Kudu provides the capability to partition hash or range partition the data for quicker access.
Isolation Levels[09]
Kudu allows for the user to set two different read operation modes through the Kudu API clients. By default, Kudu uses the 'Read Committed' isolation level. For the 'Snapshot Isolation' level the timestamp can either be set explicitly by the user or assigned by a server. The 'Snapshot Isolation' read level allows for consistent and repeatable reads.
Joins[06][13]
Joins just with Kudu are not supported. However when used with a query engine like Impala, Kudu tables can be joined with other tables stored in the Hadoop storage system.
Logging[07][14][11]
Kudu employs a logical log of operations (insert/update/delete). The operation logs are replicated for each tablet (partitions of the database). An operation log for each tablet is stored separately from the physical data. Operation logs are not able to be replayed or shipped. For single row operations (on a single tablet) Kudu is ACID. However for multi-row operations (on a single tablet), the Atomic property is not fully preserved since a single failed write will not rollback the entire operation, so per-row errors are possible. Multi-tablet operation are currently no possible with Kudu.
Query Compilation[07]
Kudu employs the use JIT compilation (using LLVM) for record projection operations specifically. The reason for this optimization is that new records are initially stored in MemRowSets, which are in-memory row-stored units of a tablet. To compensate for the lower performance of the row-stored nature of these units JIT compilation is used for projection operations that may touch these new records.
Query Execution[07]
Kudu executes both reads and writes on batches of rows which are actually columnar in-memory (as they are on disk). If there are predicates in the query, lazy materialization is used.
Query Interface[15]
Kudu has No-SQL client APIs for C++, Java and Python. Kudu can also be used with SQL-based query processing interfaces like Hadoop's Impact, MapReduce and Spark.
Storage Architecture[11]
Although, an experimental version of Kudu does rely on persistent memory in a blocked cache, Kudu is primarily disk-oriented.
Storage Model[03]
Because Kudu is designed primarily for OLAP queries a Decomposition Storage Model is used. Kudu's columnar data storage model allows it to avoid unnecessarily reading entire rows for analytical queries. The strong-type (immutable type) requirement for the columns also allow for compression of a single data type.
System Architecture[16][17]
Kudu is designed for distributed work tasks as a result the system architecture is a shared-nothing master-slave architecture. The database is horizontally partitioned (so each row is in the same partition) into tablets. Each tablet is replicated (typically 3 or 5 times) and each of these instance is stored on its own tablet server. Each tablet server has multiple unique tablets. For each tablet, one of the servers at which the tablet is stored is the leader and the rest of the servers are the followers. Leaders are elected using Raft consensus. Reads for a specific tablet can be done from any one of the tablet servers that store that tablet. Writes are only sent to the leader tablet server and whether the operation is accepted or not is determined by Raft consensus.
Citations
17 sources- Apache Kudu - Fast Analytics on Fast Data apache.org
- GitHub - apache/kudu: Mirror of Apache Kudu · GitHub github.com
- https://kudu.apache.org/docs apache.org
- Apache Kudu - Wikipedia wikipedia.org
- Apache Kudu - Using Apache Kudu with Apache Impala apache.org
- Apache Kudu - Overview apache.org
- https://kudu.apache.org/kudu.pdf apache.org
- https://kudu.apache.org/docs/schema_design.html#encoding apache.org
- Apache Kudu - Transaction Semantics in Apache Kudu apache.org
- Next-Generation Big Data: A Practical Guide to Apache Kudu, Impala, and Spark - Butch Quinto - Google Books google.com
- Apache Kudu - Kudu FAQ apache.org
- Apache Kudu - Apache Kudu Schema Design apache.org
- https://www.quora.com/Does-Clouderas-new-storage-engine-Kudu-support-joins quora.com
- Apache Kudu Transaction Semantics | 5.14.x | Cloudera Documentation cloudera.com
- Apache Kudu - Introducing Apache Kudu apache.org
- https://www.cloudera.com/documentation/enterprise/latest/topics/kudu_concepts_architecture.html cloudera.com
- Latest Insights on Data and AI | Cloudera Blog cloudera.com