Airflow Meetup (November 16, 2016)

WePay will be hosting the next Airflow meetup in Redwood City.

Talks:

Training Machine Learning Models with Airflow
Speaker: Matthew Chen, Staff Software Engineer @ WePay

WePay is the leading provider of payments-as-a-service for online platforms including GoFundMe, FreshBooks, and Constant Contact. More than 17,000 tasks flow through Airflow every day at WePay. We'll discuss how we use Airflow to power our data warehouse's ETL pipeline, and how we integrated Airflow with Google cloud platform, BigQuery, and Google cloud storage.

Migrating from Luigi to Airflow
Speaker: Alex Van Boxel, Committer on the Google Cloud part of Airflow @ Vente-Exclusive.com (Belgium)

Vente-Exclusive.com is a big online retail shop (flash sales) focused on Northern Europe. Because we have a small BI team we need a decent workflow manager. At the start Luigi worked out well but as our workflow grew it had some downsides. Since then Airflow popped up and had so many must-have features that we decided to migrate to Airflow. This talk brings the story of the migration and why we did it.

Spark with Airflow
Speakers: Rob Froetscher, Contributor of Airflow EMR tooling, Data Engineer at Lumos Labs; Tim Chan, Data Architect at Lumos Labs

At Lumos Labs (maker of Lumosity) we use Spark to crunch through massive amounts of gameplay data and generate personalized insights for our users. We use Airflow to coordinate this work, and increasing amounts of our other batch data processing. We'll run through how we use Airflow to do this (we've contributed our tooling to the Airflow master branch), and why we think it's a good choice.

Schedule:

6:30-7:00 Meet, greet, and eat.
7:00-8:00 Talks
8:00-8:30 Updates on the project, roadmap & upcoming Airbnb sprints, and Q/A for the core project team.
8:30-9:00 Community open mic, step up and make announcements if you're recruiting, looking for help, planning on working on a feature, ...