NYC Apache Airflow Meetup

• Speaker: Shrividya Hegde, Airflow Champion & Sr. AI Data Engineer

In this session, we'll explore how Apache Airflow — the battle-tested orchestration platform already running in your data stack — has evolved to become a first-class runtime for AI workloads. We'll dig into the six hardest problems of AI orchestration (reliability, cost control, provider lock-in, human oversight, observability, and integration complexity) and show exactly how Airflow and the new apache-airflow-providers-common-ai package address each one.

You'll leave understanding how Airflow's durable execution model eliminates wasted API spend on retries, how HookToolset turns 350+ existing provider hooks into AI agent tools with zero new authentication setup, how built-in Human-in-the-Loop review gates bring enterprise governance to agentic workflows, and why Airflow's provider-agnostic connection layer lets you switch between OpenAI, Anthropic, Bedrock, and 20+ other models without touching a single DAG.

We'll also compare Airflow's approach to alternatives like LangChain, CrewAI, Prefect, and AWS Step Functions — and make the case that Airflow isn't competing with AI frameworks; it's the production floor they should run on.

Talk #2: Airflow DAG Failure Investigator

• Speaker: Anton Paljusevic, Data Engineer at Third Point

Debugging a failed DAG run means context-switching across tools: scanning logs in the Airflow UI, grepping through S3, searching GitHub.

This session walks through an agent that automates that workflow.
The agent fetches task logs from S3, extracts root-cause errors, and cross-references recent PRs and upstream Airflow issues to surface whether the failure is a known regression, a provider bug, or something in your own code. The result is a plain-language report that tells you not just what broke, but why.

This talk covers the architecture, prompting strategies for extracting signal from noisy logs, and lessons learned from real production failures.

Talk #3: Skills vs. Pipeline: Two Ways to Build the Same AI Workflow

• Speaker: Vikram Koka, Chief Strategy Officer at Astronomer & Airflow Committer & PMC Member

When AI starts hallucinating your project reports, you have a choice: prompt harder, or rethink the architecture. In this talk, Vikram Koka, Chief Strategy Officer at Astronomer and Apache Airflow PMC member, shares what happened when he tried to automate his own job — tracking progress across major Apache Airflow Improvement Proposals using Claude, GitHub, and Confluence.

He built it first as an AI skill. It worked, until it didn't. Wrong counts, editorializing, self-contradiction. So he rebuilt the same workflow as a 12-task deterministic pipeline with structured Pydantic outputs, a validation LLM, and arithmetic checks. The hallucinations stopped.

This session walks through both architectures side by side — the token economics, the accuracy tradeoffs, and the practical question every AI builder eventually faces: when do you let the agent decide, and when do you take back control?

AGENDA

• 5:30-6 PM: Arrivals, networking, food & drinks
• 6-7:45PM: Presentations
• 7:45-8PM: Networking