Best practices with Airflow- an open source platform for workflows & schedules

Schedule:

6:30-7:00PM Mixer
7:00-7:35PM Presentation
7:35-8:00PM Q&A

Background:

Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

http://photos1.meetupstatic.com/photos/event/a/7/a/1/600_442602913.jpeg

Presenter:

Maxime Beauchemin recently joined Airbnb as a Data Engineer developing tools to help streamline and automate ETL processes. He mastered his data warehousing fundamentals while at Ubisoft and was an early adopter of Hadoop/PIG while at Yahoo! in 2007. More recently at Facebook he developed "analytics as a service" frameworks around engagement and growth metrics computation, anomaly detection, and cohort analysis. In his free time, he`s a father of 3 and a digital artist. More about his projects on his blog http://mistercrunch.blogspot.com/