RG Treffen Datamonsters Münsterland 2025/10 - ADF & Databricks

Azure Data Factory (ADF) is Microsoft’s central orchestration and integration service for reliable and scalable data movement. Combined with Azure Databricks as a powerful platform for data processing and transformation, it enables a flexible and future-proof data engineering architecture.
In this session, participants will gain a practical, real-world overview of how to implement complex end-to-end data loading processes with ADF and Databricks - whether on-premises, in the cloud, or in hybrid scenarios.
We will cover the full lifecycle of a Data Factory and its integration with Databricks:

· Architecture & best practices for high-performance, scalable, and maintainable pipelines
· Source and target connectivity (SQL Server, Azure Storage, APIs, Data Lake, and more)
· Databricks integration within ADF pipelines as the central transformation tool
· Transformations with Databricks notebooks – from simple cleansing to complex business logic
· Parameterization & reusability of pipelines, datasets, and notebook executions
· DevOps integration: Building a complete CI/CD pipeline in Azure DevOps or GitHub Actions
· Infrastructure as Code with Bicep and Terraform for repeatable, parameterized deployments
· Monitoring & error handling – from alerting to automated remediation

The session guides participants through a real-world end-to-end solution based on an actual customer project. The focus is on practical architecture decisions, proven best practices, and common pitfalls - rather than abstract concepts.
Participants will leave with a clear, actionable roadmap that enables them to successfully implement similar data integration scenarios with Azure Data Factory, Azure Databricks, and DevOps-based deployment.