Skip to content

Details

Overview
As AI agents move from prototypes to production, evaluation must become a continuous lifecycle rather than a one-time benchmark. From defining success criteria and generating realistic test scenarios to measuring safety, compliance, recommendation quality, and monitoring performance over time, organizations need evaluation frameworks that evolve alongside their AI systems.
This session explores the AI evaluation lifecycle through the case study of a production retail shopping assistant. It covers how persona-driven conversation simulation, LLM-as-judge metrics, and risk-aware score aggregation can be combined into an automated evaluation pipeline that supports development, regression testing, and CI/CD deployment. The session also introduces SERA (Suite of Evaluation Metrics for Retail Agents), a framework for evaluating agent behavior across Safety, Compliance, and Agent Quality, with a particular focus on multi-turn conversations where users change preferences, introduce adversarial requests, or test privacy boundaries.
Attendees will gain practical guidance on designing an end-to-end evaluation strategy for agentic AI, identifying failure modes before deployment, and operationalizing continuous evaluation throughout the AI development lifecycle. While the examples focus on a retail shopping assistant, the principles and evaluation patterns apply broadly to production AI agents across industries.

Agenda - always use this standard agenda (unless we are hosting in a new location, or hosting a virtual event)
6:00pm - 6:30pm: Networking - food & drinks
6:30pm - 8:00pm: Presentation & discussion
8:00pm: Networking at Barrel Theory

Meet Our Featured Speaker
Sowmya Podila is a Lead Applied AI Scientist at Target with more than 10 years of experience in AI/ML across AWS, Gartner, and Target. She led Target's TrendBrain initiative, featured in Retail Dive and CNBC for using AI to elevate the style and design of Target's owned brands. She co-authored SimRetail, a multi-agent system for retail assortment planning accepted at AAMAS 2026, and speaks regularly on agentic AI, evaluation, and LLM systems.

Sponsors
This event is hosted by Lab651 & Recursive Awesome

Related topics

Events in St Paul, MN
Artificial Intelligence
Artificial Intelligence Applications
Computer Vision
Internet of Things (IOT)
Technology Startups

You may also like