Mastering Apache Airflow®: Automation, Scalability & Real-World Success Stories

Details

Join fellow Airflow enthusiasts and industry leaders at the Marriott Downtown for an evening of insightful talks, great food and drinks, and exclusive swag!

This event brings together seasoned pros to explore cutting-edge strategies for managing complex DAG codebases, orchestrating large-scale data workflows, and unlocking the powerful new capabilities of Apache Airflow 3. Don't miss this opportunity to connect, learn, and level up your Airflow expertise!

PRESENTATIONS

Contending with Complex DAG Codebases - a Codemodding Approach

Speaker: Alex Tomic, Data Engineering Independent Contractor
As Airflow adoption continues to increase, organizations are faced with a mounting challenge of maintaining projects over the long-term, while meeting the demand for new project onboarding. Codemodding, a process to automatically modify code, is a valuable new tool to empower project and platform teams to develop, manage and support ever more complex dag codebases and workflows.

Harmonizing Music Rights at Scale: Airflow for Ownership Resolution and Fair Royalties

Speaker: Pei-Chi Chen, Lead Data Engineer, Create Music Group
In this session, we’ll share how Create Music Group leverages Apache Airflow to resolve music rights ownership conflicts—an ongoing challenge that increases costs and disrupts operations. We’ll dive into our Airflow-driven approach for large-scale, vital data collection and discuss best practices we’ve adopted from Airflow Summit 2024 to combat zombie tasks, out-of-memory issues, and worker configuration tuning. We'll also showcase how these techniques have helped us distribute revenue more accurately and streamline daily operations. This session provides a real-world, production-level example of Airflow’s power in action.

Unlocking the Future of Data Orchestration: Introducing Apache Airflow® 3

Speaker: Constance Martineau, Senior Product Manager, Astronomer
Airflow 3 is here, bringing a new era of flexibility, scalability, and security to data orchestration. This release introduces powerful new capabilities to simplify pipeline management, enhance security, and enable seamless execution across diverse environments. In this session, we’ll explore key innovations, including an upgraded UI, DAG Versioning, scheduler-integrated backfills, stronger task isolation, and support for remote execution. Join us to discover how Airflow 3 is redefining orchestration and empowering teams to build and scale workflows like never before!

AGENDA