Introduction to Apache Airflow: author, schedule and monitor workflows

Apache Airflow is one of the most popular tools for implementing data engineering, machine learning, and DevOps workflows, and it is used by more than 200 companies such as Airbnb, Yahoo, Paypal, Intel or Stripe.

Airflow allows to programmatically author, schedule and monitor workflows, which are sequence of tasks, started on a schedule or triggered by an event, and it is frequently used to handle big data processing pipelines.

We are going to work with Airflow from scratch and write our first DAG, accordingly to:
- airflow operators,
- operators that we write by ourselves,
- our own pipelines, using a small operator.

If time permits, we are also going to find out how to add operators using MongoDB operators.

What to bring:
This is an IN-PERSON hands-on workshop, so you will need to bring your own laptop.

Requirements:
Follow the instructions to install some packages and let's work as much as possible! https://gitlab.com/ebuitragod/airflow-playground/-/blob/master/paso-a-paso.txt

Speaker: Esperanza Buitrago
Level: Intermediate
Language: English

***

• Access Control (IMPORTANT!)

The security access control requires an attendee list. Provide your full name (first name and surname) in your profile, otherwise you will be removed from the list.

• What to bring
This is a hands-on workshop, so you will need a laptop.

• Please remember our attendance policy: https://www.meetup.com/PyLadies-BCN/messages/81834174/.

In short:

• If you are a man with a non-male friend attending the meetup: join the meetup and indicate which PyLady invited you

• If you do not have any friend that can add you as their +1, don’t worry! RSVP and send a brief message to the meetup organizer explaining why are you interested in attending this particular meetup.

• Code of Conduct
This event is for everybody, so feel free to bring your friends! For security reasons they will need to join the meetup, so they appear in the attendee list. Please keep in mind our Code of Conduct: http://www.pyladies.com/CodeOfConduct/