Meetup @ Spotify: Google Cloud Dataflow via Scio & Google Bigtable Learnings

Hey folks!

I (Jake Ouellette, Software Engineer @ Twitter), am now organizing the Data Engineering Meet-up! I'm super excited about this month's meet-up. We'll be hosting at Spotify / Echonest in Somerville.

Event will roughly be 6:30-8:30. We'll start with networking/food, followed by two awesome talks by Spotify engineers (A 15 minute one and a [TBD (>25, <60 minute)] length one!), with Q&A and discussion following.

Neville Li - Neville is a software engineer at Spotify who works mainly on data infrastructure and tools for machine learning and advanced analytics. In the past few years he has been driving the adoption of Scala and new data tools for music recommendation, including Scalding, Spark, Storm and Parquet. Before that he worked on search quality at Yahoo! and old school distributed systems like MPI.

Talk: Learn about Scio, a Scala API for Google Cloud Dataflow (incubated as Apache Beam). Beam introduces a simple, unified programming model for both batch and streaming data processing while Scio brings it much closer to the high level API many data engineers are familiar with. We will cover design and implementation of the framework, including features like type safe BigQuery and REPL. There will also a live coding demo.

Owen Heneghan - Owen is a software engineer at Spotify working on user understanding. He has helped build pipelines for aggregations of user information across various segments of musical identity. He helps create and maintain services to expose user information and preferences. Before Spotify's acquisition of The Echo Nest, he worked on a variety of tasks, including customer ingestions, web apis, and data aggregation.

Talk: Google Bigtable on GCP as a data store! Intro to Bigtable concepts and various lessons we've learned from our iterations on using Bigtable to store data about Spotify user listening behavior.

Special thanks to our sponsors:

Event venue and Beer graciously sponsored by Spotify. (special thanks to Bo Williams @ spotify for organizing)

Soda / Food are excellently being sponsored by Twitter.

Please let me know if you're interested in hosting a Data Engineering meet-up in your office, or would be interested in speaking at a future event!