History
SciDB emerged from the [Extremely Large Data Base (XLDB) Conference] (https://www.xldb.org/about/) first hosted in 2007. The conference was organized by the [SLAC National Accelerator Laboratory’s] (https://www6.slac.stanford.edu/) Scalable Data Systems team to address the gap between current database systems and the needs of data-intensive scientific projects such as the [Large Synoptic Survey Telescope (LSST) astronomical survey] (https://lsst.slac.stanford.edu/). Dave Dewitt and Mike Stonebraker agreed to lead the development of a database that would fulfill the needs of these projects. A SciDB workshop was hosted at the second XLDB conference in 2008 and code development began the same year. In 2009, Mike Stonebraker and Marilyn Matz co-founded [Paradigm4] (https://www.paradigm4.com/). Paradigm4’s team developed SciDB into a robust commercial software product and continue to develop and improve the two offered versions of SciDB: an open-source, Community Edition, and a proprietary, Enterprise Edition that offers additional functionality and customer-specific solutions.
- http://doi.org/10.2481/dsj.7.88
- http://www.odbms.org/wp-content/uploads/2014/04/The_Architecture_of_SciDB.pdf
- https://www.xldb.org/about/
- https://www6.slac.stanford.edu/research/scientific-programs
- https://lsst.slac.stanford.edu/
- https://www.lsst.org/
- https://web.archive.org/web/20130510204033/http://www.vldb.org/pvldb/2/vldb09-76.pdf
- https://www.theregister.co.uk/2010/09/13/michael_stonebraker_interview
- https://blog.goldenseeds.com/how-did-she-do-it-a-q-a-with-marilyn-matz-ceo-and-co-founder-of-paradigm4-2d196d95c83b
- https://www.crunchbase.com/organization/paradigm4
- https://www.paradigm4.com/
- https://www.paradigm4.com/scidb-is-available-in-two-editions/
Compression
Run-Length Encoding Null Suppression
SciDB allows users to define how each attribute of an array will be compressed when the array is created. The default is no compression. The additional options are zlib, bzlib, or null filter (null suppression) compression. Since SciDB stores data by attribute, vertically partitioning logical chunks of an array into single-attribute physical chunks, the specified compression is used on a chunk-by-chunk basis. If certain parts of a chunk are accessed more often than others, causing overhead due to decompression and recompression, SciDB can partition a chunk into tiles and compress on a tile-by-tile basis. Run-length encoding is used to compress recurring sequences of data. In addition, SciDB’s storage manager compression engine can split or group logical chunks in order to optimize memory usage while remaining within the limit of the buffer pool’s fixed-size slots.
Website
Source Code
https://forum.paradigm4.com/t/index-of-scidb-releases/773
Developer
Paradigm4
Country of Origin
Start Year
2008
Project Type
Academic, Commercial, Open Source