How Oracle manages Airflow in research & production & highlights on Airflow 2.0


Szczegóły
**This is a free online meetup via Zoom - must RSVP to get the link.
*This event will be in English!
How Oracle manages Airflow as a service for research and production data teams:
In this meetup session, we will have a special guest speaker, Gita Ferber, Oracle’s Senior Software Developer. Gita will dive deeply into Apache Airflow at Oracle: Airflow for production and Airflow for research teams. Get a sneak peek into how Oracle built out their projects, so you can gain some tips and best practices to implement on your own Airflow project.
Airflow for production:
Oracle’s use case and why they use Airflow
Working with Airflow over time: how their architecture changed
Airflow in large scale: how they run Airflow cluster with over 30k tasks
Automating the CI/CD process
Airflow orchestration as a service for research teams:
The different requirements for research
Running heavy research pipelines using Airflow + DBND open source
Using multiple compute environments
Airflow then and now:
Evgeny, Databand’s co-Founder and CTO, will then discuss the pains that Airflow 1.0 has that Airflow 2.0 will solve.
Apache Airflow 2.0 highlights and deep dive:
Functional DAGs
Airflow Scheduler (HA and Performance )
Serialized DAGs and Versioned Dags
KubernetesExecutor and KEDA
Packaging
What's the current status? What’s missing? What’s Next?

Sponsorzy
How Oracle manages Airflow in research & production & highlights on Airflow 2.0