History[06][07]

Deep Information Sciences was founded in 2010 based on research conducted at the University of New Hampshire. After the company went under in 2017, the source code of the DeepDB engine was released as open-source as part of a new Deep Software Foundation holding. A large portion of the source code of the system was a custom C++ implementation of the Java Development Kit software and not related to the DBMS itself.

Checkpoints[08][09][10]

DeepDB supports last transaction checkpoints, and it takes checkpoints asynchronously.

Compression[11]

There exists prefix compression in indexes. DeepDB keeps compressed data in the cache, and decompress it during operations. The system supports high-levels of compression with a compact representation of keys and delta compression.

Concurrency Control[11][09]

DeepDB utilizes Multi-version Concurrency Control such that the database can be rolled back to any transactional state.

Data Model[12]

While the DeepDB storage engine implements a Key/Value model, the data model is fully relational, as specified in MySQL.

Indexes[09][12][13]

The index DeepDB uses does not belong to any choice. It uses hyper-indexing and the new CASSI(Continuous Adaptive Sequential Summarization of Info) tree for indexing, as a replacement for B+Tree or LSM-Tree type indexing. CASSI tree is a persistent data structure that supports ACID transactions. It is improved from B+Trees in that they are able to collapse the internal structure through virtualizing and summarizing. CASSI tree provides O(1) write complexity and O(log(N)) read complexity in the worst case. Based on the type of data and tasks, whether transactional, data stream capture, or analytics, the tree and indexes used in queries are dynamically adjusted to maximize hardware resources. For speed of lookup, sometimes it chooses to index every column in a database table.

Joins[14]

DeepDB is a storage engine full compatible with other components in MySQL. Thus, it supports the same join algorithms as MySQL, including the Nested-Loop Join Algorithm and the Block Nested-Loop Join Algorithm.

Logging[09]

DeepDB maintains state logs so that the replicas can generate the same indexes and perform the same operations from the streamed state log.

Query Interface[08][15]

DeepDB is designed as a storage engine that is compatible with MySQL. Since it uses the same MySQL API, the query interface is also the same, which supports the standard SQL query interface with several additional extensions.

Storage Architecture[16][11][17]

DeepDB runs on POSIX-based disk-oriented file-systems (e.g. SSD/HDD). All files are persistently stored on the disk and it will write data into in-memory files temporarily. DeepDB stores data in 3 forms including on-disk row store tables, in-memory row store tables, and on-disk column store indexes. Instead of organizing in pages, the in-memory row store is designed to manage single rows as much as possible. The data and indexes on disk in memory are organized into segments, with various sizes. Segments may contain summary data or metadata so that metadata or summary data remain in the cache when the segments are evicted.

The system manages cache usage using adaptive algorithms. Variable-sized segments rather than pages are used to store data. In addition, summary indexing is used to identify relevant segments.

Storage Model[08][09]

DeepDB supports transactional row storage for OLTP workloads. It also implements segmented column stores to deliver capabilities similar to OLAP databases (column store). Different from traditional column stores that maintain large and monolithic column store files, variable-sized segments are maintained in DeepDB to improve space utilization.

Storage Organization[16]

A shadow copy is maintained for recovery on the system crash. Disk snapshots are maintained to roll forward and back in data history. The database files including indexes, transactional state, and metadata are streamed append-only files.

System Architecture[16][03][18]

DeepDB storage engine is designed as an easy-to-install plugin replacement for MySQL's native InnoDB storage engine. Using DeepDB does not require any application or schema change. It augments MySQL with full ACID compliance and additional machine-learning metrics. The system is architected for complex environments and supports HTAP(Hybrid Transactional Processing).