22th Eindhoven Data Community Meetup | Studyportals

We are happy to announce our 22nd Eindhoven Data Community meetup which we host together with Studyportals at there beautiful office at the Klokgebouw!

We've got an exciting evening planned, diving into dbt orchestration and deployment.

First up, Katie and Armand from Studyportals will share their journey to "Just-In-Time Data," revealing how they optimized dbt runs for 450 models and over 30 sources to deliver data by 9 AM, evolving their setup to a dynamic lineage in Airflow.

After a short break, Koen Graat from Xebia Data will explore "How to use CI/CD pipelines for dbt and deploy dbt Cloud using Infrastructure as Code," with a sneak peek into dbt Fusion. Learn how to revolutionize your dbt deployments, enhance collaboration, and improve data quality.

The Journey to Just-In-Time Data: Evolving our dbt Orchestration with Airflow
We strive for our dbt project to be ready by 9am for our stakeholders. Should be easy, right? Except that our dbt project consists of around 450 dbt models and over 30 sources. Some of those sources are ready as early as midnight but some as late as 4am, and in total our project takes around 4 hours to run.
Join as us we walk through the evolution of our dbt run setup, from one selector, to a set of parallel commands, to today's setup -- a dynamic lineage in Airflow which runs models when and only when the upstream source is ready. It's finished when the Tableau datasource is refreshed and our stakeholders can start their day with the latest data.

How to use CI/CD pipelines for dbt and deploy dbt Cloud using Infrastructure as Code + sneak peak into dbt Fusion
Are you tired of manual and failing dbt deployments? This talk explores how CI/CD and IaC can revolutionize your data transformation workflows, enhancing collaboration and data quality within your dbt projects. Learn the core concepts of CI/CD, including automated testing and deployment pipelines, I will guide you through building a CI/CD pipeline for dbt, triggering it with code changes and running comprehensive tests. Next to that we will dive into Infrastructure as Code (IaC) and how it automates dbt Cloud deployments using tools like Terraform. You will gain practical knowledge for automating dbt Cloud resources, projects, and environments. As a bonus we will do a sneak peek into the recently announced dbt Fusion engine.

Location: Studyportals, Klokgebouw 280, Eindhoven

Program:

• 17:00 – 18:00 Food
• 18:00 – 18:10 Welcome
• 18:10 – 19:00 Katie + Armand: The Journey to Just-In-Time Data: Evolving our dbt Orchestration with Airflow
• 19:00 – 19:15 Break
• 19:15 – 20:00 Koen: How to use CI/CD pipelines for dbt and deploy dbt Cloud using Infrastructure as Code + sneak peak into dbt Fusion
• 20:00 - 21:00 Drinks

About Katie Scheitzer | Senior Analytics Engineer | Studyportals
Katie is a Senior Data Analytics Engineer, who has more than 10 years of experience working in data, inlcuding web analytics, data science, and analytics engineering. She is originally from the US, but came to the Netherlands to get her Masters in Data Science and has been working here since.

About Armand Duijen | Data Engineer | Studyportals
Armand is a Data Engineer from Eindhoven, working at Studyportals. He has nearly five years of experience building and maintaining data pipelines in the AWS cloud. With a background in Computer Science from TU/e, he also explores machine learning to help Studyportals predict student behavior. Outside of work, he enjoys running and exploring new places.

About Koen Graat | Principal Data Engineer | Xebia Data
Koen is a Data Engineer with experience in various roles, including data engineering, team leadership, and data architecture. He brings an entrepreneurial mindset and a focus on delivering business value through data solutions. Even though he has a master in Data Science, he quickly transitioned after into data engineering and has since worked on data platforms and architectures for large organizations in banking, energy, and trading. Koen is a strong advocate for dbt and data engineering best practices, and he is passionate about bridging the gap between business needs and technological solutions, and enjoys knowledge sharing.