Building robust data pipelines with dbt, Airflow, and Great Expectations

Hi PyLadies Berlin! We're back with another great event for you!🙌

🔎 Description
Data quality has become a much discussed topic in the fields of data engineering and data science, and it has become clear that ensuring data quality is absolutely crucial to avoiding a case of "garbage in - garbage out". Apache Airflow and dbt (data build tool) are some of the most prominent tools in the data engineering ecosystem, and while dbt offers some data testing capabilities, enhancing the pipeline with data validation through Great Expectations can add additional layers of robustness.

This talk will outline a convenient pattern for using these tools in what we've been calling the "dAG stack": Build a transformation layer and test those transformations with dbt, validate the source data and add more complex tests as well as data documentation with Great Expectations, and orchestrate the entire pipeline with Airflow. You'll see some examples of how the tools fit together and complement each other in order to build a robust data pipeline.

📆 Agenda
19h00 Communities introduction
19h10 Non-coding superpower: With journaling to a more intentional and organized life by Larissa Haas
19h30 Talk + Q&A with Sam Bail
20h30 See you next time! \o/

🎙 About the Speakers

Sam Bail is a data professional with a passion for turning high quality data into valuable insights. Sam holds a PhD in Computer Science and has worked for several data-focused startups. In her current role as Engineering Director at Superconductive, she works on “Great Expectations”, an open source Python library for data validation and documentation.

Larissa Haas is a Data Scientist working at sovanta, located in Heidelberg. Besides wrangling with data, she is interested in reading/writing Science Fiction (a lot), robots gone rogue and Ultimate Frisbee.

---
• By attending our online event, you agree to the PyLadies Code of Conduct: https://www.pyladies.com/CodeOfConduct/

• Contact
Interested in speaking at one of our events? Have a good idea for a Meetup? Get in touch with us at berlin@pyladies.com

Find us on the PyLadies Global workspace:

1. https://slackin.pyladies.com enter your email address.
   Accept the email invitation
2. Go to workspace https://pyladies.slack.com
3. Join channel #city-berlin, #germany, #jobs-europe