Airflow Meetup @ Paypal
Details
Hello Airflow community,
Paypal is hosting Airflow meetup !!
Here are more details.
Presentations & Talks:
- Paypal using Airflow to remediate Live Hadoop cluster to keep it highly available.
Speaker: Jayesh Senjaliya, Software Engineer @ Paypal
Description: Paypal has one of the biggest Hadoop clusters in the industry. ~ It's challenging to keep up with sporadical hardware failures to keep the cluster highly available. Also its humanly impossible to manually take care of bad nodes and replace with new ones at such a large scale. So Paypal takes a workflow based approach to automate the entire process to remediate nodes on live Hadoop clusters.
- Predictive Analytics with Airflow and PySpark
Speaker: Russell Jurney, Principal Consultant, Data Syndrome
Description: Russell will talk about how to wrangle PySpark into a full blown analytics application using Airflow.
- Advanced Data Engineering patterns with Apache Airflow
Description: Analysis automation and analytic services is the future of data engineering! Apache Airflow's DSL makes it natural to build complex DAGs of tasks dynamically, and Airbnb has been leveraging this feature in intricate ways, creating a wide array of services as dynamic workflows. In this talk, we'll explain the mechanics of dynamic pipeline generation using Apache Airflow, and present advanced use cases that have been developed at Airbnb.
Speaker: Maxime Beauchemin, Airbnb
- Panel discussion on 2017 Roadmap for Apache Airflow
Speakers: Apache Airflow PMC members
...
Schedule :
5:30-6:30 Meet, greet, eat, and drink
7:00-8:00 Talks
8:00-8:30 Q&A, Updates, Roadmap
8:30-9:00 Community open-mic, updates, etc.
Airflow Meetup @ Paypal