Airflow in the Wild: K8s, DAGs, and Operational Zen

Join fellow airflow enthusiasts and leaders at Samba TV's Office for an evening of engaging presentations, delicious food and drinks, and exclusive new Airflow t-shirts!

From Samba’s K8s setup to Astronomer’s self-aware pipelines, real Airflow lessons from real Airflow teams.

PRESENTATIONS

Samba Airflow: The Good, The Bad, and The Ugly

• Speaker: Piotr Kostrzeński, Software Engineer at Samba TV
• Running Apache Airflow on Kubernetes isn’t always smooth sailing. In this talk, Piotr Kostrzenski walks through Samba’s Airflow deployment, from initial setup to production challenges. You'll hear what went well, what broke spectacularly, and how the team adapted their infrastructure to support large-scale data workflows. It’s a candid, behind-the-scenes look at the realities of orchestrating Airflow in a containerized world.

How the Airflow Company Uses Airflow

• Speaker: Steven Hillion, SVP of Data & AI at Astronomer
• Astronomer’s data team recently underwent a major transformation in how we use Apache Airflow. In response to several operational challenges, we re-architected our workflows to improve reliability, scalability, and observability. This shift involved adopting dataset scheduling and breaking down our pipelines into micro-pipelines to reduce failure rates and boost overall stability. We also implemented robust observability tools to manage complex dependencies and gain full end-to-end visibility into our pipelines. To streamline onboarding and support growth, we standardized the use of Task Groups across our workflows. With Airflow now managing itself more effectively, we’re able to refocus on delivering value through data rather than wrestling with infrastructure. As a testament to these improvements, we’ll be sharing some of our favorite stats from the terabyte of data we process daily—offering a unique look into how data teams around the world are leveraging Airflow.

Dynamic DAGs in Airflow

• Speaker: Clayton Cole, Senior Data Engineer at Samba TV
• In this talk, Clayton Cole will explore how to support and scale Dynamic DAGs in Apache Airflow. As data teams grow and workflows become more complex, the need for flexibility and automation in DAG generation becomes critical. Clayton will walk through key patterns, best practices, and lessons learned from building dynamic pipelines that adapt to changing data, configurations, and operational needs. Whether you're new to Dynamic DAGs or looking to level up your implementation, this session will provide practical insights and real-world examples to help you make the most of Airflow’s dynamic capabilities.

AGENDA

• 5:30-6PM: Arrivals, eat, drink, and network
• 6-7:40PM: Presentations
• 7:40-8:30PM: Networking