Apache Airflow is a leading framework for managing data workflows. It is easy to learn the basics, it is extremely flexible, it enables tech teams to programmatically author, schedule, and monitor their workflows, and it brings everything together in an intuitive and convenient UI. It's also open source.
It started at Airbnb, and organisations like PayPal and Yahoo! have quickly adopted it.
This is a group for anyone who wants to learn about or share what they know about Apache Airflow. We think it's a really exciting tool and we want to help grow the community.
Talk #1: Let it Whirl - Make Airflow Development & Testing Fun again
Speaker: Bas Beelen, Data Juggler at GoDataDriven
Developing Airflow DAGs can be hard. What if there's only one Airflow instance running both production and work-in-progress DAGs? What if you have restricted access to Airflow due to security policies? What if you are extracting part of your logic to a separate Python module, hence working simultaneously in two projects? How do you prevent from going crazy and how do you make Airflow development fun again?
He & his colleague Kris Geusebroek struggled with such situations the past six months at ING Wholesale Banking Advanced Analytics. To make Airflow DAG development fun and productive again, they invested time in locally developing and running these DAGs. Since they are probably not the only ones experiencing these difficulties, he would like to share with us the approach they took.
Talk #2: Introduction to Google Cloud Composer & Demo
Speaker: David Sabater, Cloud Customer Engineer, Big Data & ML at Google
David will give a brief introduction to Cloud Composer, the Apache Airflow managed service in Google Cloud Platform. We will then run a demo showing how to transfer data between the US and EU in a scalable way.
Note: Please use your Full Name as the reception requires it to create your badge.