NYC Apache Airflow Meetup

PLEASE NOTE- YOU MUST REGISTER HERE TO JOIN

Join Apache Airflow committers, users, and community leaders for our Monthly Airflow Town Hall! This one-hour event is a collaborative forum to explore new features, discuss AIPs, review the roadmap, and celebrate community highlights.

The Town Hall happens on the first Friday of each month (unless there is a holiday) and will be recorded for those who can't attend. Recordings will be shared on the Airflow Youtube Channel and posted to the #town-hall channel on Airflow Slack and the dev mailing list.

Agenda

• Arrivals & Introduction
• Airflow Project Update
• PR of the Month Highlight
• Project Spotlight
• Community Spotlight
• Closing Remarks

PLEASE NOTE- YOU MUST REGISTER HERE TO JOIN

• Speaker: Shrividya Hegde, Airflow Champion & Sr. AI Data Engineer

In this session, we'll explore how Apache Airflow — the battle-tested orchestration platform already running in your data stack — has evolved to become a first-class runtime for AI workloads. We'll dig into the six hardest problems of AI orchestration (reliability, cost control, provider lock-in, human oversight, observability, and integration complexity) and show exactly how Airflow and the new apache-airflow-providers-common-ai package address each one.

You'll leave understanding how Airflow's durable execution model eliminates wasted API spend on retries, how HookToolset turns 350+ existing provider hooks into AI agent tools with zero new authentication setup, how built-in Human-in-the-Loop review gates bring enterprise governance to agentic workflows, and why Airflow's provider-agnostic connection layer lets you switch between OpenAI, Anthropic, Bedrock, and 20+ other models without touching a single DAG.

We'll also compare Airflow's approach to alternatives like LangChain, CrewAI, Prefect, and AWS Step Functions — and make the case that Airflow isn't competing with AI frameworks; it's the production floor they should run on.

Talk #2: Airflow DAG Failure Investigator

• Speaker: Anton Paljusevic, Data Engineer at Third Point

Debugging a failed DAG run means context-switching across tools: scanning logs in the Airflow UI, grepping through S3, searching GitHub.

This session walks through an agent that automates that workflow.
The agent fetches task logs from S3, extracts root-cause errors, and cross-references recent PRs and upstream Airflow issues to surface whether the failure is a known regression, a provider bug, or something in your own code. The result is a plain-language report that tells you not just what broke, but why.

This talk covers the architecture, prompting strategies for extracting signal from noisy logs, and lessons learned from real production failures.

Talk #3: Airflow Meets LangChain and LlamaIndex: Production AI Pipelines Without the Black Box

• Speaker: Vikram Koka, Chief Strategy Officer at Astronomer & Airflow Committer & PMC Member

AI frameworks like LangChain and LlamaIndex solve the reasoning problem brilliantly. They do not solve the production operations problem. When an agent makes 15 LLM calls, hits a rate limit on step 5, and retries from scratch, all you see from the outside is: "agent ran."

This talk introduces the outer loop / inner loop architecture in Apache Airflow's common.ai provider, which wraps AI framework logic in observable, independently retryable Airflow tasks. We will walk through a live SEC 10-K financial analysis pipeline built with LlamaIndex and LangChain: how Dynamic Task Mapping creates per-company retrieval tasks at runtime, how Human-in-the-Loop gates enforce analyst approval before delivery, and how structured Pydantic output and token usage limits replace manual cost controls. Each step that would be invisible inside agent.run() becomes a named task with logs, retry behavior, and an audit trail.

Attendees will leave with a practical mental model for decomposing any AI pipeline into governed Airflow tasks, and a working architecture they can apply to their own LangChain or LlamaIndex workloads.

AGENDA

• 5:30-6 PM: Arrivals, networking, food & drinks
• 6-7:45PM: Presentations
• 7:45-8PM: Networking