Cloud BigTable
BigTable[1] is a distributed storage system used in Google, it can be classified as a non-relational database system. BigTable is designed mainly for scalability. It typically works on petabytes of data spread across thousands of machines.
There is not much public information about the detail of BigTable, since it is proprietory to Google. The most authoritative information about it is its paper[1]. An open source implementation of it based on its original paper is Apache HBase[2].
Google has now provided BigTable as its cloud NoSQL database service[3]. The documentation of that[4] might be helpful, too.
History
BigTable was among the early attempts Google made to manage big data. Jeffrey Dean and Sanjay Ghemawat were involved in it. It is one of the three components Google built for managing big data (the other two are Google File System[1] and MapReduce[2]).
These three components focus on different aspects of big data: Google File System is a reliable distributed file system that the other two build upon; MapReduce is a distributed data processing framework; BigTable is a distributed storage system.
These three projects are very famous in distributed system. They all have their open source implementation.[3][4][5]
- 1 http://static.googleusercontent.com/media/research.google.com/zh-CN//archive/gfs-sosp2003.pdf
- 2 http://static.googleusercontent.com/media/research.google.com/zh-CN//archive/mapreduce-osdi04.pdf
- 3 https://hbase.apache.org/
- 4 https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html
- https://hadoop.apache.org/
Storage Model
In BigTable, a table is split into multiple tablets, each of which is a subset of consecutive rows[1]. A tablet is a unit of data distribution and load balancing. Different tablets of a table may be assigned to different tablet servers. A tablet is stored in the form of a log-structured merge tree[2] (which they call memtable and SSTable).
Furthermore, BigTable allows clients to create locality group[3]. A locality group is a subset of columns in a table. BigTable will create a separate SSTable for each locality group, which will improve read performance of this locality group.
- 1 http://static.googleusercontent.com/media/research.google.com/zh-CN//archive/bigtable-osdi06.pdf section 2
- 3 http://static.googleusercontent.com/media/research.google.com/zh-CN//archive/bigtable-osdi06.pdf section 6
- 2 O’Neil
- Patrick
- et al. The log-structured merge-tree (LSM-tree). Acta Informatica 33.4 (1996): 351-385.
Data Model
BigTable does not support relational data model. Instead, it provides users the ability to create column families in a table.
Each table usually contains a small number of column families, which should be rarely changed (because the change of them involves metadata change). Inside each column family, there can be unlimited number of columns. Users can freely add or delete columns in a column family. Deleting of an entire column family is also supported.
BigTable does not have any type information associated with a given column. It only treats data as strings of bytes.
Storage Architecture
BigTable assumes an underlying reliable distributed file system (here is Google File System). The tablets are stored in Google File System, which is a disk-oriented file system. The most recently written records are stored in memtable, which is in memory. However, most of the data is stored on disk.
Website
https://cloud.google.com/bigtable/
Developer
Country of Origin
Start Year
2005