Storage Organization

Indexed Sequential Access Method (ISAM)

Accumulo, which Sqrrl is built on top of, uses a file called a Relative Key File (RFS) which is a type of Indexed Sequential Access Method (ISAM). These hold data from tablets, which in Accumulo are partitions of tables.

System Architecture

Shared-Nothing

With native Hadoop integration, Sqrrl Enterprise ingests data. It is encrypted and labeled, then indexed and stored in Sqrrl's version of Apache Accumulo within the Hortonworks Data Platform (HDP). Sqrrl and HDP, together, store multi-structured data in its native format. This allows users of Sqrrl to make queries of many different kinds. Sqrrl is considered a distributed database.

Checkpoints

Consistent

Based on the Accumulo implementation, checkpoints are organized with the Hadoop Distributed File System (HDFS) in order to create a system of of redundant files. The replication framework creates two instances of Accumulo that are consistent with each other.

Compression

Delta Encoding Bit Packing / Mostly Encoding

Based on the implementation of Accumulo, Sqrrl likely uses several types of compression. GZip and LZO are two compression algorithms used by Accumulo. In addition, Accumulo uses delta encoding by resending information if it is the same as prior sent information. Delta encoding is also used when storing sequential information in the table rows.

Concurrency Control

Multi-version Concurrency Control (MVCC)

Concurrency control is implemented in Sqrrl in a way that is not publicly known. Sqrrl supports concurrency control by doing atomic updates on data entities. Sqrrl is also built on top of Zookeeper, which has locks to control concurrency.

Data Model

Key/Value Graph

After data is ingested, the Sqrrl Enterprise product details each piece at a cell level. Data can be a key/value pair or a field in a JSON document. Sqrrl then uses secondary indexing techniques to store the data in Apache Accumulo in its native form. This flat architecture allows Sqrrl to handle data of different structures.

In 2016, a presentation by Sqrrl stated that the version of Accumulo used by Sqrrl Enterprise stores sorted key-value pairs. Sqrrl then combines the data into graph storage form, using a linked data model.

Sqrrl is therefore considered a graph DBMS because it both represents data to users in a graphical format and is schema-free. Custom iterators for Accumulo developed by Sqrrl allow users to complete graph analysis on the data.

Foreign Keys

Not Supported

Accumulo does not have built-in foreign keys as a NoSQL DBMS. The specific implementation in Sqrrl is not publicly known.

Isolation Levels

Read Committed

Based on the Accumulo implementation, isolation can be activated. When it is, a user reading the database will either see none of the updates while they are being applied to a row, or they can see all of the updates after they are applied to the row. However, at this level of isolation, users may see different values from a row on two different reads, or have phantom new rows be added that are only visible after the reading transaction is over.

Logging

Logical Logging

The exact kind of logging Sqrrl uses is not known, but based on the Accumulo documentation, it likely saves each write to its log files.

Query Compilation

Code Generation

Sqrrl converts SQL queries into Accumulo iterators.

Query Interface

Custom API

Sqrrl Enterprise offers users a proprietary query language called SqrrlQL that is integrated with the cell-level security concept. Users are able to execute SQL-like queries (key-value), full-text queries, or graph searches.

Storage Architecture

Hybrid

Based on the Accumulo database management system Sqrrl is built on, Sqrrl uses both disk space in the form of Hadoop and Zookeeper locks and an in-memory store of sorted key/value pairs.

Storage Model

Custom

Sqrrl is considered to be a wide column store, which is a kind of NoSQL storage model.

Within Accumulo, which Sqrrl is built on, partitions of tables (called tablets) organize the columns and values within a row boundary so that the information for any particular row is in one tablet.