NYC Airflow Meetup at Astronomer HQ!
Details
Join fellow Airflow enthusiasts and leaders at Astronomer HQ for an evening of engaging talks, great food and drinks, and exclusive swag!
We’re especially excited to welcome Apache Airflow PMC Member Vikram Koka who will be visiting from the Bay Area—don’t miss the chance to meet him in person and hear him speak!
PRESENTATIONS
Talk #1: Airflow + AI : Orchestrating the Intelligent Stack
- Speaker: Shrividya Hegde, Airflow Champion & Sr. AI Data Engineer
In this session, we'll explore how Apache Airflow — the battle-tested orchestration platform already running in your data stack — has evolved to become a first-class runtime for AI workloads. We'll dig into the six hardest problems of AI orchestration (reliability, cost control, provider lock-in, human oversight, observability, and integration complexity) and show exactly how Airflow and the new apache-airflow-providers-common-ai package address each one.
You'll leave understanding how Airflow's durable execution model eliminates wasted API spend on retries, how HookToolset turns 350+ existing provider hooks into AI agent tools with zero new authentication setup, how built-in Human-in-the-Loop review gates bring enterprise governance to agentic workflows, and why Airflow's provider-agnostic connection layer lets you switch between OpenAI, Anthropic, Bedrock, and 20+ other models without touching a single DAG.
We'll also compare Airflow's approach to alternatives like LangChain, CrewAI, Prefect, and AWS Step Functions — and make the case that Airflow isn't competing with AI frameworks; it's the production floor they should run on.
Talk #2: Airflow DAG Failure Investigator
- Speaker: Anton Paljusevic, Data Engineer at Third Point
Debugging a failed DAG run means context-switching across tools: scanning logs in the Airflow UI, grepping through S3, searching GitHub.
This session walks through an agent that automates that workflow.
The agent fetches task logs from S3, extracts root-cause errors, and cross-references recent PRs and upstream Airflow issues to surface whether the failure is a known regression, a provider bug, or something in your own code. The result is a plain-language report that tells you not just what broke, but why.
This talk covers the architecture, prompting strategies for extracting signal from noisy logs, and lessons learned from real production failures.
Talk #3: Airflow Meets LangChain and LlamaIndex: Production AI Pipelines Without the Black Box
- Speaker: Vikram Koka, Chief Strategy Officer at Astronomer & Airflow Committer & PMC Member
AI frameworks like LangChain and LlamaIndex solve the reasoning problem brilliantly. They do not solve the production operations problem. When an agent makes 15 LLM calls, hits a rate limit on step 5, and retries from scratch, all you see from the outside is: "agent ran."
This talk introduces the outer loop / inner loop architecture in Apache Airflow's common.ai provider, which wraps AI framework logic in observable, independently retryable Airflow tasks. We will walk through a live SEC 10-K financial analysis pipeline built with LlamaIndex and LangChain: how Dynamic Task Mapping creates per-company retrieval tasks at runtime, how Human-in-the-Loop gates enforce analyst approval before delivery, and how structured Pydantic output and token usage limits replace manual cost controls. Each step that would be invisible inside agent.run() becomes a named task with logs, retry behavior, and an audit trail.
Attendees will leave with a practical mental model for decomposing any AI pipeline into governed Airflow tasks, and a working architecture they can apply to their own LangChain or LlamaIndex workloads.
AGENDA
- 5:30-6 PM: Arrivals, networking, food & drinks
- 6-7:45PM: Presentations
- 7:45-8PM: Networking
