Observability, Testing, and Trust for Agentic AI with MLflow
Details
REGISTER AT THE LUMA EVENT PAGE!!!
https://luma.com/2e6rhzjo
Agenda:
6:00 - 6:30 PM - Welcome and mingle
6:30- 6:45 PM - Introductions
6:45 - 7:30 PM - Talk
7:30 - 8:00 PM - Wrap up
Description:
IAI coding agents can now scaffold entire applications from a single prompt. But for mission-critical systems that require compliance, reliability, and auditability, "it works on my machine" isn't enough. When your agent makes fifty tool calls, rewrites three files, and confidently hallucinates a security vulnerability, how do you even begin to debug that?
This talk introduces a practical, open-source toolkit for injecting observability, control, and systematic testing into agentic AI development.
Using MLflow's tracing and evaluation framework, we'll follow a single throughline: building an AI agent with a coding assistant (Claude Code), tracing both the assistant's work and the agent it produces, then evaluating quality with LLM judges - closing the loop from development through production monitoring.
You'll walk away with a concrete workflow for turning the black box of agentic coding into something you can inspect, measure, and trust. No vendor lock-in required.
Speaker Bio:
Tim Lortz is an AI Product Specialist at Databricks, where he has worked since 2019. He currently serves as the technical lead for Databricks' go-to-market efforts for AI in regulated industries, focusing particularly on enterprise-grade governance for AI platforms.
Prior to Databricks, Tim spent many years in various data science practitioner and leadership roles in the Federal contracting space. He holds a PhD in Industrial & Operations Engineering from the University of Michigan (Go Blue!!) and a BS in Industrial Engineering from the University of Pittsburgh.
