Databricks Workflows CICD and Automated Testing

Please join us on 22nd May, 2024 to listen to the topic: Databricks Workflows CICD and Automated Testing

What ~ Toronto Data Professionals Community (Virtual)
When ~ Wednesday, 22nd May, 2024

Agenda:

• 6:00 PM Networking and Introduction
• 6:15 PM Topic: Databricks Workflows CICD and Automated Testing with Dustin Vannoy
• 7:30 PM End

Where: Online via Microsoft team

Session Details:
Databricks Workflows (also known as Jobs) are a great choice for automating data pipelines. Once the code is ready comes the important step of promoting beyond your dev environment. Continuous Integration / Continuous Deployment (CI/CD) involves versioning, testing, and deploying your data processing jobs. Databricks provides tools that allow us to follow these DevOps best practices, but how do we put these together to ensure quality and manage workflow promotion across isolated environments? Join this session to learn some of the most common ways teams leverage Databricks to version, test, and deploy their automated data pipelines. In this session we cover some basic CI/CD concepts and the options within Databricks. Then we walk through an example of merging, testing, and deploying a workflow change.

Speaker Bio:
Dustin Vannoy is a Data Engineering Consultant experienced in solving business problems with analytics and big data solutions. He is passionate about all aspects of data engineering, especially building data platforms and streaming data pipelines. He currently focuses on building data platforms and pipelines in Apache Spark / Databricks, Kafka, Python, and Scala. He is co-founder of the Data Engineering San Diego meetup and encourages others to grow their data skills by making tutorials, mentoring others, and speaking at events.