Apache Airflow: Author, Schedule and Monitor Data Workflows

Description:
Apache Airflow is a tool created by the community to programmatically author, schedule and monitor workflows. The biggest advantage of Airflow is the fact that it does not limit the scope of pipelines. Airflow can be used for building Machine Learning models, transferring data or managing the infrastructure. During this talk, Jarek - Apache Airflow Committer and PMC member, will provide an introduction to Airflow. The lecture will cover the most important components of Airflow: Directed Acyclic Graphs that define workflows, task operators, integration hooks, task executors and the scheduler. Jarek will also talk about best practices for debugging DAGs and implementing custom operators. Finally, he will provide a guide to when Airflow is the right choice and when other solutions should be considered.

Jarek will also dive into details of the modern development environment he helped to improve for the one of the most active Apache project - with a distributed team of over a hundred contributors.

Jarek Potiuk, Polidea - Enthusiastic and pragmatic Engineer, Technology Evangelist, Software Gardener, geek and gadget lover. Intelligent, extremely good in problem investigation and solving, presenting high integrity, sharing passion, enthusing others. With a strong personality, but at the same time a team player. Focused on mobile, cloud and everything in-between. Experienced in Robotics, with a good understanding of AI context. Founder of Mobile Warsaw and MCE. Speaker at Mobile Warsaw, DevFest Poland 2016, Codepot, Agile Warsaw and multiple other events.