The Snowflake Elastic Data Warehouse

"The Snowflake Elastic Data Warehouse", by Dageville, Cruanes, Zukowski, et al (2016)

Presented by Michael Jalkio

Abstract:

We live in the golden age of distributed computing. Public cloud platforms now offer virtually unlimited compute and storage resources on demand. At the same time, the Software-as-a-Service (SaaS) model brings enterprise-class systems to users who previously could not afford such systems due to their cost and complexity. Alas, traditional data warehousing systems are struggling to fit into this new environment. For one thing, they have been designed for fixed resources and are thus unable to leverage the cloud’s elasticity. For another thing, their dependence on complex ETL pipelines and physical tuning is at odds with the flexibility and freshness requirements of the cloud’s new types of semi-structured data and rapidly evolving workloads.

We decided a fundamental redesign was in order. Our mission was to build an enterprise-ready data warehousing solution for the cloud. The result is the Snowflake ElasticData Warehouse, or “Snowflake” for short. Snowflake is a multi-tenant, transactional, secure, highly scalable and elastic system with full SQL support and built-in extensions for semi-structured and schema-less data. The system is offered as a pay-as-you-go service in the Amazon cloud. Users up-load their data to the cloud and can immediately manage and query it using familiar tools and interfaces. Implementation began in late 2012 and Snowflake has been generally available since June 2015. Today, Snowflake is used in production by a growing number of small and large organizations alike. The system runs several million queries per day over multiple petabytes of data.

In this paper, we describe the design of Snowflake and its novel multi-cluster, shared-data architecture. The paper highlights some of the key features of Snowflake: extreme elasticity and availability, semi-structured and schema-less data, time travel, and end-to-end security. It concludes with lessons learned and an outlook on ongoing work.

http://info.snowflake.net/rs/252-RFO-227/images/Snowflake_SIGMOD.pdf

Please note that the venue for this meetup has changed, to Ugather Cowork in Hillcrest. Everything else about the event will be as usual! Pizza will still be provided.

Parking tips:

* Do not park in the HUB or Cali-O-Burgers parking lot! They Tow.
* Parking is generally easy to find on Cleveland Ave, Normal Street, Pennsylvania Ave (4 blocks south).
* Remember that parking meters go until 8pm in Hillcrest

If you're interested in presenting a paper please fill out this form (https://docs.google.com/forms/d/e/1FAIpQLScaI-fWdys27-ByT_HdtsJ73V4AxZr0hf1GSqLsQ1IwAaPdIQ/viewform) or talk to us in person at the meetup.