TileDB is an efficient multi-dimensional array management system which introduces a novel on-disk format that can effectively store dense and sparse array data with support for fast updates. It offers numerous features, including excellent compression, high IO performance on multiple data persistence backends (e.g., HDFS and S3), and easy integration with ecosystems used by today’s data scientists (e.g., Python NumPy).[04]
- Website
- http://tiledb.org[01]
- Source Code
- https://github.com/TileDB-Inc/TileDB[02]
- Developers
- Country of Origin
- US
- Start Year
- 2017 [05]
- Project Types
- Commercial, Open Source
- Written in
- C++
- License
- MIT License
TileDB is an efficient multi-dimensional array management system which introduces a novel on-disk format that can effectively store dense and sparse array data with support for fast updates. It offers numerous features, including excellent compression, high IO performance on multiple data persistence backends (e.g., HDFS and S3), and easy integration with ecosystems used by today’s data scientists (e.g., Python NumPy).[04]
History[05]
TileDB was originally created at the Intel Science and Technology Center for Big Data, a collaboration between Intel Labs and MIT. The research project was published in a VLDB 2016 paper. TileDB, Inc. was founded in February 2017 to continue the further development and maintenance of the TileDB software.
Compression
TileDB offers a variety of compressors to choose from: GZIP, Zstandard, LZ4, RLE, Bzip2, Double-delta TileDB also implements its own version of double-delta compression. It is similar to the one presented in Facebook’s Gorilla system. The difference is that TileDB uses a fixed bitsize for all values (in contrast to Gorilla’s variable bitsize). This makes the implementation a bit simpler, but also allows computing directly on the compressed data (which we are exploring in the future).
GenomicsDB
Citations
6 sources- http://tiledb.org tiledb.org
- https://github.com/TileDB-Inc/TileDB github.com
- https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17_TileDB.pdf mit.edu
- https://docs.tiledb.io/en/stable/index.html tiledb.io
- Academy • TileDB tiledb.io
- https://tiledb.io/about tiledb.io