AI Governance & Engineering Best Practices on Databricks

Welcome to the Manchester Databricks Meetup - March 2026

Join us for an evening packed with Databricks insights, real-world use cases, expert speakers, and plenty of time to network with the local data & AI community.

Agenda:

17:30- 17:45 Arrival & Networking
Meet fellow data professionals, practitioners, and Databricks enthusiasts

17:45 - 18:00 Opening Remarks & Introductions
Opening remarks from the organisers

18:00- 20:00 - Sessions
Hear from industry leaders sharing practical Databricks and AI insights.
Pizza and refreshments will be served during the sessions.

20:00 onwards: Drinks & Networking
Continue the conversations over drinks and informal networking

Session 1:

Allan Doolan - Data Engineering Leader
Luca Zugic - Lead Data Engineer
Deren Ridley - Data Engineer

Your Databricks Pipelines Deserve Unit Tests: True Local-First TDD Using pytest and the Databricks Runtime Docker Image

Join Allan, Luca, and Deren as they explore a challenge many Databricks teams face: unit testing PySpark pipelines.
Too often, teams skip proper testing because PySpark feels awkward to test locally. We decided not to.
In this session, we’ll show how we built a local-first, test-driven development (TDD) workflow for PySpark pipelines using pytest, the Databricks Runtime Docker image, and a set of pragmatic engineering patterns.

They'll cover:

• Intelligent Spark session management that automatically detects your environment
• Fast and reliable DataFrame assertions using PySpark’s built-in testing utilities
• How running tests inside the same Docker image as production gives you confidence — without needing a cluster

Expect live code, real patterns from a production codebase, and a look at how we’re integrating Claude into our Data Engineering workflow to accelerate development.
If you build data pipelines on Databricks and want faster feedback loops, safer deployments, and cleaner engineering practices, this talk is for you.

Session 2 - James Bentley - Senior AI Consultant

Go With the MLflow: Untangling “Spaghetti AI” with the Databricks AI Governance Framework

As many organisations rush to adopt AI, they often find themselves with a fragmented ecosystem of tools, copilots, and models that are difficult to manage, monitor, or govern.

Join James to learn how he helped organisations tackle exactly this problem — transforming a tangled landscape of “Spaghetti AI” into a clean, unified, and fully governed AI platform powered by Databricks.
During the session, James will share how the Five Pillars of AI Governance became the blueprint for bringing structure and control to the AI lifecycle. He’ll also explore how the team is:

• Standardising AI development using open-source frameworks
• Centralising LLM activity through Mosaic AI
• Implementing MLflow Experiments for production monitoring
• Creating long-term auditability and regulatory readiness across AI use cases

The result is a move away from fragmented AI experiments toward a secure, observable, and scalable enterprise AI platform.

Date and Location:
Date: 19th March 2026

Venue: Slalom – Floor 15, Bloc, 17 Marble Street, Manchester, M2 3AW