Skip to content

Details

Modern data platforms demand more than just scalable storage and fast pipelines — they require trustworthy, validated data at every stage. As organizations adopt Microsoft Fabric to unify data engineering, analytics, and governance, the need for automated, repeatable, and code‑driven data quality controls becomes critical.
In this session, we will dive deep into how Great Expectations, a widely‑used open‑source data validation framework, can be integrated directly into Microsoft Fabric Lakehouses, Warehouses, and Data Pipelines to enforce data quality rules at scale. We’ll explore:

  • How Great Expectations operates inside Fabric Notebooks and connects to tables.
  • Patterns for embedding validation into Fabric Data Pipelines using checkpoints and orchestration.
  • Techniques for designing robust, parameterized expectation suites to validate schema, statistical distributions, referential integrity, and business rules
  • How to store, version, and operationalize validation artifacts using Fabric’s Git integration
  • Approaches for recording validation results and exposing them through DQ observability dashboards in Fabric or Power BI
  • Common performance and architecture considerations when running GE in distributed Spark environments within Fabric

This session is ideal for data engineers, architects, and advanced BI practitioners who want to implement production‑grade data quality frameworks within Fabric. Expect real demos, practical patterns, and examples you can apply immediately in your own environment.

Related topics

Microsoft Azure
Data Center and Operations Automation
Python
Computer Programming
DevOps

Sponsors

McGill DataSphere Lab

McGill DataSphere Lab

Hosting the meetup group once a month at McGill University.

You may also like