What we’re about
Apache Airflow is a leading framework for managing data workflows. It is easy to learn the basics, it is extremely flexible, it enables tech teams to programmatically author, schedule, and monitor their workflows, and it brings everything together in an intuitive and convenient UI. It's also open source.
It started at Airbnb, and organisations like PayPal and Yahoo! have quickly adopted it.
This is a group for anyone who wants to learn about or share what they know about Apache Airflow. We think it's a really exciting tool and we want to help grow the community.
See here for more: https://airflow.apache.org/
Sponsored by https://www.astronomer.io/
Upcoming events (2)
See all- Bangalore Apache Airflow® Meetup at Cloudera!Cloudera, Bengaluru
Meet up with fellow Airflow users at Cloudera's Bangalore Office for an afternoon of awesome talks and networking.
This is the perfect opportunity to get to know like-minded Airflow users, partners, and company experts, and learn how customers are utilizing Airflow within their organizations.
🔥 Agenda
- 1:30 PM – 2:00 PM: Arrival & Settling In
- 2:00 PM – 2:30 PM: Talk 1 – The Evolution of Orchestration: Airflow’s Journey and It’s AI-Powered Future
- Speaker: Shubham Raj Software Engineer II - CDE at Cloudera & Shreya Sareen Product Manager at Cloudera
- 2:30 PM – 3:00 PM: Talk 2 – Breaking the Connection: Why Airflow Tasks No Longer Touch the Database
- Speaker: Amogh Desai, Senior Software Engineer at Astronomer & Apache Airflow PMC Member
- 3:00 PM – 3:30 PM: Talk 3 – GenAI based log summarization and mitigation steps at Uber
- Speaker: Sameer Raj & Atish Anand, Uber
- 3:30 PM – 4:00 PM: Networking & Snacks
🎤 Featured Talk Highlights
Talk 1- The Evolution of Orchestration: Airflow’s Journey and It’s AI-Powered Future
Speakers: Shubham Raj & Shreya Sareen
Workflow orchestration has been on quite a journey. It started with tools, which gave data teams a way to manage complex Hadoop jobs at scale. Then came Airflow, bringing Pythonic DAGs, a flexible scheduler, and an active community quickly becoming the go-to platform for modern data pipelines. Now, a new chapter is opening: AI is beginning to reshape how we design and interact with workflows. From helping generate DAGs to making orchestration more accessible, AI is pushing us toward a more intuitive developer experience.
This talk traces that evolution, from foundational tools to Airflow’s rise, and the AI-powered frontier ahead.Talk 2- Breaking the Connection: Why Airflow Tasks No Longer Touch the Database
Speaker: Amogh Desai, Senior Software Engineer, Astronomer & Apache Airflow PMC Member
Airflow tasks traditionally had direct access to the internal database — a design that caused scalability, security, and dependency issues. With AIP-72, Airflow introduces a radical improvement: running tasks in isolated environments that communicate with Airflow via a lightweight API. Learn how this new approach enables:- Better security with isolated execution
- Elimination of dependency conflicts
- Task execution on any infrastructure
- Simplified development experience
Talk 3 - Piper (an Apache Airflow fork) serves hundreds of teams at Uber, with ~200k pipelines, and 800k tasks run in a day.
Speakers: Sameer Raj & Atish Anand, Uber
With huge scale comes the huge responsibility of supporting users, with >300 support queries a week. This huge inflow of queries is overwhelming for the on-call engineer to tackle, leading to delays in resolution. The queries, for example, can be like "Why did my task fail?", "Why was my pipeline paused?", "Can you help me with writing a task to upload a file to HDFS?", etc.
Thus, to solve this, we leveraged GenAI and developed agents.
The support resolution rate is 45%.
*** - Network event119 attendees from 21 groups hostingAirflow Summit 2025Hyatt Regency Seattle, Seattle, WA
Join us at the Airflow Summit in Seattle from October 7-9, 2025, for a deep dive into the future of data with Airflow 3.0. Connect with over 500 professionals, explore 100+ expert-led sessions, and participate in hands-on workshops to expand your knowledge and skills in the Apache Airflow ecosystem. Don’t miss out on this exciting event to shape the next era of data workflows!
PLEASE NOTE-YOU MUST REGISTER HERE TO SECURE A TICKET
Visit the Airflow Summit website to get your ticket and learn more!
PLEASE NOTE-YOU MUST REGISTER HERE TO SECURE A TICKET
Past events (16)
See all- Network event228 attendees from 26 groups hostingBeyond Analytics: The DataOps ConferenceThis event has passed
