SWC-DB

View Current Viewing Revision #4 from 07/25/2021 3:01 p.m.

SWC-DB ("Super Wide Column Database") is a DBMS with support for super-wide key.

The key is supporting 16777216(2^24,uint24) number of Fractions, max 2^24 bytes a Fraction and total size of a key is up-to 2^32 bytes.

SWC-DB does not have Tables nor Namespaces whereas the number of columns is a numerous (at date max of unsigned 64 bit integer).

SWC-DB uses a proprietary SQL (structural query language) that suit the requirements to work with it's super wide-key.

Compression

Naïve (Page-Level) Naïve (Record-Level) Bit Packing / Mostly Encoding

The available encoders (compressors) are ZSTD, ZLIB, SNAPPY and in PLAIN. On a record-level base configuration it is done by updating Cell-Value with the desired encoder. On a page-level base that is applied at Cell-Store-Block it is done by defining the default configuration for a schema or by column-schema definition with the encoder to use. Transactions of Communications are configurable at a program specific (application level) with an application designated configuration property and file. Variable-Length integers are used for most of numerical primitives either long-term/persistent storage or intermediate run-time consumption.

Concurrency Control

Multi-version Concurrency Control (MVCC) Deterministic Concurrency Control Timestamp Ordering

The option to define the maximum cell-versions to keep/retain in SWC-DB is done by configuring the column's Schema property "cell versions", whereas the order and the revision of a cell is determined by the Cell's definitions of timestamp and it's order DESC/ASC. The timestamp defaults to AUTO that is swcdbRanger application system's time and order to ASC. Additionally to the user-supplied timestamp SWC-DB cell has an internal cell-revision definition that let to determine the latest data that should be retain or removed.

The precision of timestamp used in SWC-DB is in nanoseconds. Whereas the configurable 'cell ttl' property of a column schema is in milliseconds. The Cell's TTL of the schema is applied against timestamp of a cell and at reach of TTL the cell is auto-considered as non-existing, depends on momentum and max-versions, by changing the Schema's cell-ttl before Compaction has under-gone It is a possibility the Cell to exist.

The File-System of SWC-DB is configurable and on current date supports Hadoop-JVM, Hadoop(Native C++), Ceph, Local, Custom (a user developed library) and FsBroker. Depends on SWC-DB release/build-type, filesystem/s can be an implementation or to be a dynamically loaded library. The FsBroker-Filesystem for SWC-DB applications is a filesystem-type whereas swcdbFsBroker(application) has an underlying fs-type configuration. The swcdbFsBroker can be a running on remote instances to the hosts running the swcdbManager/s or/and swcdbRanger/s. The Custom-Filesystem is a type name-holder for a user developed File-System library based on SWC::FS::FileSystem interface class (limitless of FS Impl. possibilities).

The SWC-DB base FS path is determined and defined by the configuration-properties --swc.fs.TYPE.path.root and --swc.fs.path.data and it is evaluated as swc.fs.TYPE.path.root="/var/opt/swcdb/" joined with swc.fs.path.data="your/cluster/name". As example to have FS-base at root folder of HDFS it is swc.fs.hadoop_jvm.path.root=/swcdb/ plus joined with swc.fs.path.data="or/no/sub/folders".

The data-topology(files/folders structure) within the SWC-DB base-path is based by column-id and range-id, that on path consist CellStores and CommitLog files while at any point one server is responsible for a range-id on column-id and of a path root. The CellStores are files storing Cells in serialized form that are after latest compaction whereas Commit-Log is many Fragments of current added data, the fragments are written on a (schema configurable log-rollout-ratio) threshold reach or on shutdown. SWC-DB uses structure ID folder-names and a single-folder won't have more than 2000 entries a folder.