Hardware Acceleration

Custom

SciDB is intended to be used with inexpensive and widely available hardware. This design decision provides users flexibility in maintaining their system, they can add nodes in order to increase capacity and/or performance, and provides users the freedom to choose the hardware that best fits their requirements.

Compression

Run-Length Encoding Null Suppression

SciDB allows users to define how each attribute of an array will be compressed when the array is created. The default is no compression. The additional options are zlib, bzlib, or null filter (null suppression) compression. Since SciDB stores data by attribute, vertically partitioning logical chunks of an array into single-attribute physical chunks, the specified compression is used on a chunk-by-chunk basis. If certain parts of a chunk are accessed more often than others, causing overhead due to decompression and recompression, SciDB can partition a chunk into tiles and compress on a tile-by-tile basis. Run-length encoding is used to compress recurring sequences of data. In addition, SciDB’s storage manager compression engine can split or group logical chunks in order to optimize memory usage while remaining within the limit of the buffer pool’s fixed-size slots.

Stored Procedures

Supported

Logging

Logical Logging

SciDB logs all queries using [Apache log4cxx] (https://logging.apache.org/log4cxx/latest_stable/).

Concurrency Control

Multi-version Concurrency Control (MVCC)

Foreign Keys

Not Supported

Foreign keys are not part of the array data model used by SciDB.

Storage Architecture

Disk-oriented

SciDB is disk-oriented, which allows it to support the large scale of data that may be stored for a single application.

Query Interface

Command-line / Shell

Indexes

Not Supported

SciDB does not use an index. Instead, it maps chunks of an array to specific nodes by hashing the chunk’s coordinates. SciDB also has a map that allows dimensions specified with user-defined data types to be represented internally as integers, which is called an index in the SciDB documentation.

Storage Model

Decomposition Storage Model (Columnar)

SciDB stores data by attribute, vertically partitioning logical chunks of an array into single-attribute physical chunks.

Data Model

Array / Matrix

SciDB supports multi-dimensional arrays. Upon creating an array, the user specifies its dimensions and attributes. Each unique set of dimensions maps to a single cell in the array. Each cell is defined by a collection of attributes, where an attribute represents a single data value. Both dimension and attribute data types can be user-defined. This provides users with the flexibility to specify coordinates and/or classifications that fit their applications. If dimensions are not specified, SciDB creates a data frame - an unordered group of cells. Users can also create temporary arrays, which are stored in-memory and do not keep deltas of changes like non-temporary arrays do.

System Architecture

Shared-Nothing

SciDB has a shared-nothing system architecture, which is intended to support the scalability of the system. Query processing occurs at each node on the data at that node. When creating an array, a user may specify the distribution of the array data: whether chunks will be stored primarily on one node or replicated on all nodes.