Airflow Meetup (June 14, 2016)

After Airflow's successful first-ever meetup, hosted by Airbnb, we're moving the second meetup south to WePay in Redwood City.

Talks:

Airflow @ WePay - Building an ETL pipeline with Airflow
Speaker: Chris Riccomini, Committer/PPMC on Apache Airflow

WePay is the leading provider of payments-as-a-service for online platforms including GoFundMe, FreshBooks, and Constant Contact. More than 17,000 tasks flow through Airflow every day at WePay. We'll discuss how we use Airflow to power our data warehouse's ETL pipeline, and how we integrated Airflow with Google cloud platform, BigQuery, and Google cloud storage.

Airflow @ Agari - A Fault-tolerant Scoring Pipeline
Speaker : Sid Anand, Committer/PPMC on Apache Airflow

Agari, a leading email security company, is applying big data and cloud best practices to the security industry to better secure the world against email-borne threats. Agari does this by collecting email telemetry data from its enterprise customers, building models, and then applying those models to score emails (i.e. on a suspicion scale). I'll discuss our cloud based architecture (AWS SNS, SQS, Kinesis, Auto-Scaling, S3, Lambda, EMR Spark, etc...) and how we use Airflow to manage and coordinate our model building and scoring pipelines.

Airflow @ Yahoo - Data Ingestion Pipeline
Speaker - Praveen Venkatesan

Flurry, acquired by Yahoo, is an industry leading mobile analytics tool. We process data for 250k developers, 800k apps and 10B sessions daily. Airflow plays a critical role at flurry in helping ingest data into druid, a distributed column oriented data store. This talk will discuss why we use airflow instead of Oozie, how we use it and things to consider when running airflow.

Schedule:

6:30-7:00 Meet, greet, and eat.
7:00-8:00 Talks
8:00-8:30 Updates on the project, roadmap & upcoming Airbnb sprints, and Q/A for the core project team.
8:30-9:00 Community open mic, step up and make announcements if you're recruiting, looking for help, planning on working on a feature, ...