DeepDB
DeepDB (also sometimes called DeepSQL near the end of the project) was a proprietary MySQL storage engine designed for OLAP and OLTP workloads. Intended to scale MySQL to large scale data operations, it utilizes adaptive data structures and machine learning algorithms to optimize transactional workloads at big data scale. Different from classic B+Tree and LSM-Tree based storage engines, DeepDB is built on top of a new tree structure, called CASSI (Continuous Adaptive Sequential Summarization of Info), which dynamically configures the database during runtime to adapt to new hardware deployments. CASSI keeps running the three steps of analysis, adaption and optimization for high efficiency. Therefore, this storage engine allows enterprises to utilize MySQL without manual configuration under new hardware settings.
- https://www.informationweek.com/applications/deepdb-general-purpose-database-for-big-data-era/d/d-id/1110716
- https://www.eweek.com/database/deep-information-sciences-releases-new-deepsql-engine
- http://misclassblog.com/databases-and-data-warehouses/deepsql-the-next-generation-of-database-optimization
History
Deep Information Sciences was founded in 2010 based on research conducted at the University of New Hampshire. After the company went under in 2017, the source code of the DeepDB engine was released as open-source as part of a new Deep Software Foundation holding. A large portion of the source code of the system was a custom C++ implementation of the Java Development Kit software and not related to the DBMS itself.
Storage Architecture
All files are persistently stored on the disk and it will write data into in memory file temporarily. DeepDB stores data in 3 forms including on-disk row store tables, in-memory row store tables, and on-disk column store indexes. Instead of organizing in pages, the in-memory row store, which is append only, is designed to manage single rows as much as possible. The data and indexes on disk in memory are organized into segments, with various sizes. Segments may contain summary data or metadata, so that metadata or summary data remain in cache when the segments are evicted.
The system manages cache usage using adaptive algorithms. Variable-sized segments rather than pages are used to store data. In addition, summary indexing is used to identify relevant segments.
System Architecture
DeepDB storage engine is designed as an easy-to-install plugin replacement for MySQL's native InnoDB storage engine. Using DeepDB does not require any application or schema change. It augments MySQL with full ACID compliance and additional machine-learning metrics. The system is architected for complex environments and supports HTAP(Hybrid Transactional Processing).
Compression
Delta Encoding Prefix Compression
There exists prefix compression in indexes. DeepDB keeps compressed data in the cache, and decompress it during operations. The system supports high-levels of compression with a compact representation of keys and delta compression.
Concurrency Control
Multi-version Concurrency Control (MVCC)
Same as MySQL, it supports Multi-version Concurrency Control.
Source Code
https://github.com/DeepFound/deep_engine
Developer
Deep Information Sciences, Inc.
Country of Origin
Start Year
2010
End Year
2017
Former Name
DeepSQL