Mastering Airflow: Local Reliability and AI Orchestration

Join us at The Loft for an exciting evening of all things Airflow! This event will feature three engaging talks showcasing real-world use cases: running Apache Airflow reliably in an on-premise setup, orchestrating AI knowledge bases, and Airflow customization.

Whether you're an Airflow beginner, expert, or just curious about data orchestration, this is the perfect opportunity to learn, network, and connect with the community. Plus, enjoy complimentary food, drinks, and some exclusive Airflow swag to take home.

AGENDA

• 5:30 PM-6 PM: Arrivals, Eat, Drink and Network!
• 6 PM-7:40 PM: Presentations!
• 7:40 PM- 8 PM: Eat more, drink more, and network!

PRESENTATIONS

Orchestrating AI Knowledge Bases with Apache Airflow

• Théo Lebrun, Senior Data Engineer & Technical Manager at Ippon Technologies USA
• Theo will present how Airflow can be leveraged to manage and update AI knowledge bases across multiple data sources. We'll dive into the architecture, demonstrate how Airflow enables efficient data extraction, transformation, and loading (ETL), and share insights on tackling challenges like data consistency, scheduling, and scalability. Whether you're building your own AI-driven systems or looking to optimize existing workflows, this session will provide practical takeaways to make the most of Apache Airflow in orchestrating GenAI solutions.

Running Apache Airflow Locally and Reliably

• Damian Shaw, Software Engineer, GTS
• As a small team in GTS, we administer Apache Airflow to merge dozens of internal and external data sources and produce daily reports—some of which are time-sensitive before the U.S. equity market opens. Due to various constraints, such as no reliance on cloud services, and data segregation, we operate entirely on onsite virtual machines. This talk explores how we meet these demands while keeping Airflow stable and adaptable to our evolving needs.

How to Customize and Extend Airflow

• Jake Roach, Associate Sales Engineer, Astronomer
• As your Apache Airflow usage matures, the need for customization becomes essential to optimize workflows, enforce best practices, and extend functionality. This session dives into how you can tailor Airflow to fit your unique requirements by creating Custom Operators and Sensors, leveraging Plugins, and implementing Cluster Policies. Whether you're looking to streamline your DAG development, enforce organizational standards, or unlock new capabilities, this talk will equip you with the knowledge to take your Airflow expertise to the next level.

Please note- this event is for individuals in the data space who have knowledge of and a passion for Apache Airflow. Guests who do not fit that criteria will not be permitted to enter the event.