(Online) Apache Airflow: DAG Design, Monitoring, Lessons from the Field
Details
This is an Online event, the Teams link will be published on the right of this page for those who have registered.
18:30: Apache Airflow in Production: DAG Design, Monitoring, and Lessons from the Field - Pradeep Kalluri
19:55 Prize Draw - Packt eBooks
Session details:
Apache Airflow is the world's most widely deployed workflow orchestration platform — but running it reliably in production requires more than just writing DAGs.
In this session, Pradeep Kalluri, a Data Engineer at NatWest Bank and Apache Airflow open-source contributor, shares practical lessons from building and operating Airflow at scale in cloud environments.
Topics covered:
- DAG design patterns that scale — avoiding common pitfalls in dependency management and task granularity
- Production monitoring and alerting — what to measure, how to set up SLA alerts, and how to debug failures fast
- Celery and Kubernetes executor deep-dives — choosing the right executor for your workload
- Lessons from open-source contribution — what contributing to Airflow taught me about how it really works under the hood
- Cloud deployment best practices — running Airflow on AWS and Azure with high availability
Whether you're just getting started with Airflow or managing complex multi-team pipelines, this session will give you actionable patterns you can apply immediately.
Speaker:
Pradeep Kalluri
Data Engineer | NatWest | Building Scalable Data Platforms
Data Engineer with 3+ years of experience building production data platforms at NatWest, Accenture, and Capgemini. Specialized in cloud-native architectures, real-time processing with Kafka and Spark, and data quality frameworks. Published technical writer on Medium, sharing practical lessons from production systems. Passionate about making data platforms reliable and trustworthy.
