Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores

The Paper
Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores

Paper Link
https://www.vldb.org/pvldb/vol13/p3411-armbrust.pdf

Abstract
Cloud object stores such as Amazon S3 are some of the largest and most cost-effective storage systems on the planet, making them an attractive target to store large data warehouses and data lakes. Unfortunately, their implementation as key-value stores makes it difficult to achieve ACID transactions and high performance: metadata operations such as listing objects are expensive, and consistency guarantees are limited. In this paper, we present Delta Lake, an open source ACID table storage layer over cloud object stores initially developed at Databricks. Delta Lake uses a transaction log that is compacted into Apache Parquet format to provide ACID properties, time travel, and significantly faster metadata operations for large tabular datasets (e.g., the ability to quickly search billions of table partitions for those relevant to a query). It also leverages this design to provide high-level features such as automatic data layout optimization, upserts, caching, and audit logs. Delta Lake tables can be accessed from Apache Spark, Hive, Presto, Redshift and other systems. Delta Lake is deployed at thousands of Databricks customers that process exabytes of data per day, with the largest instances managing exabyte-scale datasets and billions of objects.

Format
We start at 6:10, don't be late!
The discussion lasts for about 1 to 1.5 hours, depending upon the paper.
• Read the paper (done before you arrive)
• Introductions (name, and background)
• First impressions (1-2 minutes this is what I thought)
• Structured review (we move through the paper in order, everyone gets a chance to ask questions, offer comments, and raise concerns)
• Free form discussion
• Nominate and vote on the next paper