Meetup Airflow Paris @Algolia

We will meet again in Mai for a meetup hosted by Algolia. This meetup is organised in collaboration with memphis.dev engineering meetup. Attention! The talks will be given in English 🇺🇸.

3 talks will take place:

Opening doors at 18h45.

1. 19h05, Orchestrating Machine Learning for Personalisation with Airflow — Algolia

Airflow is the de-facto tool when it comes to data pipelines. As the industry is moving to integrate ML at scale for production scenarios, we will dig into how Airflow can be used in a ML operations (MLOps) context, and how it can leverage capabilities offered by Cloud providers such as Google Cloud Platform. We will also talk about the challenges of personalisation using ML such as :

• dealing with imperfect data and variety across customers:
• creating a self-service, reliable solution for customers big and small
• continuously upgrading ML models in our training and inference pipelines
• ensuring predictions freshness

2. 19h35, Feed Apache Airflow with Quality Data — [Memphis.dev](http://memphis.dev/)

Abstract: Your organization is growing. New internal clients, consumers, interfaces, integrations, and APIs are joining by the day. Each one of them is controlled by a different service, a different team, a different format, and the hardest one - a different model. The path towards upstream breaks, consumer crashes, unreliable data, and disparate data engineers is set. Enforcing a standard at that point in time is complicated and almost impossible as your architecture is probably way too coupled, thats why it is crucial to lay the foundations for data governance from day 1, especially to ensure your Airflow jobs correction. In this talk, you will learn what are the risks of ingesting data without data governance, how to create a supportive framework to control your different data clients using common open-source tools, and to make sure you feed Airflow with quality data.

3. 20h, Short horror story about Composer — Artefact

This talk will be a deep dive into Cloud Composer. Indeed, we recently faced a major incident on the orchestrator of the data platform. We will share the post-mortem with you. We will also share with you the best practices to implement on Cloud Composer and give you an overview of the limitations you need to have in mind when you choose Cloud Composer.

4. Networking and drinks