Airflow Meetup @ Monzo

Save the date!! Let's meet at the Monzo office for an evening of great talks with pizza, healthy snacks, and drinks!

***

Talk #1: Airflow at Monzo: Evolving our data platform as the bank scales

Speakers:
Jonathan Rainer (Data Platform Engineer @ Monzo)
Ed Sparkes (Data engineer @ Monzo)

As a bank Monzo has seen exponential growth in active users, from 1.6 million in 2019 to 5.8 million in 2022. At the same time, the number of data users and analysts has expanded from an initial team of 4 to 132. Alongside this growth, our infrastructure and tooling have had to evolve to deliver the same value at a new scale. From an Airflow installation deployed on a single monolithic instance, we now deploy atop Kubernetes and have integrated our Airflow setup into the bank’s backend systems. This talk charts the story of that expansion and the growing pains we’ve faced and looks to the future of our use of Airflow. We’ll first discuss how data at Monzo works, from event capture to arrival in our Data Warehouse, before assessing the challenges of our Airflow setup. We’ll then dive into the re-platforming required to meet our growing data needs and some of the unique challenges that come with serving an ever-growing user base and need for analysis and insight.

***
Talk #2: Building and deploying LLM applications with Apache Airflow

Speaker: Michael Gregory (Staff Ecosystem Engineer @ Astronomer)

In the excitement surrounding Large Language Models (LLMs), very little is being said about operationalising LLM applications and the data pipelines supporting them. This talk will discuss the development and operationalization of an LLM-powered application. It shows a real-life and open-source example, Ask Astro, that illustrates how to overcome the challenges of day-2 operations with LLMs. We’ll provide a recipe for building a production knowledge interface and insights into the synergy between LLM operations and Apache Airflow, taking LLM applications from prototypes to production with Apache Airflow.

***

Thanks to Monzo for hosting the event!

***

We are looking for speakers for future Meetup Sessions. Please fill out https://forms.gle/ES1YDE6wsHy95xKf8 if you are interested.