Serverless Orchestration with Airflow: Maximizing Concurrency, Minimizing Cost

Apache Airflow is a powerful tool for orchestrating data workflows, but how do you use it effectively with serverless cloud tools to optimize both performance and cost? This talk will walk through building a DAG that dynamically analyzes incoming data size and assigns it to the right processing pipeline—ensuring efficient resource use without overprovisioning. We'll also explore how this pattern can scale to hundreds of workflows with just a simple loop, demonstrating Airflow’s flexibility in handling complex orchestration at scale. Whether you're new to Airflow or looking to refine your approach, you'll gain practical insights into maximizing concurrency while keeping costs under control.

Does your application need computer vision support? Contact our sponsor Applied Photons for a free consultation.

About Michael Pankiewicz:
"Throughout my career as a data analyst, scientist and engineer, my goal in everything I do is to look at how this helps tie into the core business. Ultimately, I strive to improve efficiency while lowering costs by maximizing resources and talent. I have built predictive models for the Pentagon, created real-time and batch processes for retail giants churning TBs at a time, and made some pretty sick dashboards if I say so myself.

As an individual, I am constantly trying to lower the time it takes to do tasks by improving myself or my coworkers. Through continued learning I've developed programming skills, conducted training sessions, cut out redundancy and taken in feedback to always build on myself."

This is an IN-PERSON event:
Greencourt Innovation Center
12358 Parklawn Dr. Rockville, MD 20852
(ground floor classroom, Wilkins Ave. side)

Agenda

• 6:30 Presentation
• 7:00 Q+A
• 7:30 Pizza & Networking

Parking

• There is free parking on-site.

Public transit

• We are conveniently located near the Twinbrook Metro station.

If you haven't done so yet, please fill out this brief, anonymous survey
so we can keep presenting topics that match your interests!