Details
WePay will be hosting the next Airflow meetup in Redwood City!
--- Schedule ---
6:30-7:00 Meet, greet, eat and drink
7:00-8:00 Talks
8:00-8:30 Project updates & roadmap, Q&A
8:30-9:00 Community open-mic, announcements, etc.
--- Direction ---
Our office is located on the 2nd floor of the building at 350 Convention Way. There will be someone at the front desk to help you badge in. If you are driving here, you can park in the parking lot across the entrance of the building. There are also street parking available as well.
--- Talk 1 ---
Title: Operating Data Pipeline with Airflow
Presenter: Ananth PackkilDurai @ Slack
Summary: Slack is a communication and collaboration platform for teams. Our millions of users spend 10+ hrs connected to the service on a typical working day. Airflow is the core system for orchestrating our data pipeline. About six months back, we started on-call rotation for our data pipeline. This talk covers the incremental steps we took to solve on call nightmares & Airflow scalability issues to make our data pipeline more reliable and simpler to operate.
--- Talk 2 ---
Title: Running Airflow inside Google Cloud Platform
Presenter: Trevor Edwards @ Google
Summary: Apache Airflow, being a platform-neutral orchestration framework, doesn't dictate how the serving infrastructure should be constructed. Google Cloud Platform (GCP), offers a portfolio of advanced, open and performant products with a focus on being an open cloud. Our product suite not only meets Airflow's infrastructure needs but also opens up many possibilities for service innovations. This talk presents a canonical way of spawning an Airflow environment with building blocks such as GKE, GCS, GAE, IAP, GCB, Stackdriver and many more. We further elaborate how these services collectively make running Airflow in GCP an ease, providing high availability, debugability and reduced operational complexity
--- Talk 3 ---
Title: How WePay runs Airflow on Kubernetes in GCP
Presenter: William Pursell @ WePay
Summary: WePay has been one of the first companies to run Airflow on K8S in GCP. Since GCP and K8S are both fast moving targets, our deployment process has been undergoing constant changes. We will discuss some of the initial design choices, lessons we learned, technical issues we currently face, and improvements that we plan to make going forward.
--- Livestream ---
Join us remote via Zoom: https://wepay.zoom.us/j/781815754
