Lookup Tables: Fine-Grained Partitioning for Distributed Databases

—The standard way to scale a distributed OLTP DBMS is to horizontally partition data across several nodes. Ideally, this results in each query/transaction being executed at just one node, to avoid the overhead of distribution and allow the system to scale by adding nodes. For some applications, simple strategies such as hashing on primary key provide this property. Unfortunately, for many applications, including social networking and order-fulfillment, simple partitioning schemes applied to many-to-many relationships create a large fraction of distributed queries/transactions. What is needed is a finegrained partitioning, where related individual tuples (e.g., cliques of friends) are co-located together in the same partition. Maintaining a fine-grained partitioning requires storing the location of each tuple. We call this metadata a lookup table. We present a design that efficiently stores very large tables and maintains them as the database is modified. We show they improve sca...

• Type: Journal
• Year: 2012
• Where: ICDE
• Authors: Aubrey Tatarowicz, Carlo Curino, Evan P. C. Jones, Sam Madden
• Keywords: Database, Database Workloads, ICDE 2012, Partition Data, Social Networking