Building a Model Evaluation Practice for Production AI

Most teams pick an AI model the same way they pick a coffee machine - the demo looks good or they hear about it from someone, so they assume it works. But in production, evaluation decisions directly impact cost, latency, accuracy, and user trust. Without a structured eval practice, you're flying blind when a model regresses, or a new release claims to be better.
This session gives you a framework for evaluating LLMs against your workloads and operationalizing evaluation as a continuous practice, not a one-time exercise.

What we'll cover:

• Why standard benchmarks are often misleading for enterprise use cases
• Building a golden dataset: how to sample, label, and version evaluation sets from real production traffic
• Key metrics beyond accuracy: hallucination rates, precision/recall tradeoffs, cost-per-correct-answer, and latency percentiles
• Running evals at scale: open-source tools (Ragas, PromptFoo, LangSmith) and using Amazon Bedrock Model Evaluation
• Setting a model evaluation gate: how to enforce a promotion threshold before any model goes to production
• Demo: running a side-by-side eval across two models on a real classification task

Speaker Bio: Udaya is a Senior Solutions Architect at AWS and an AI-DLC Champion, with 18+ years of experience designing and building enterprise applications. He provides strategic guidance to ISV, Enterprise, SMB, and Retail customers on becoming AI-First organizations, focusing on three pillars: (1) AI-DLC - accelerating how you build software using the AI-Driven Development Lifecycle, delivering exponential velocity and quality across the entire development pipeline; (2) Agentic Products - embedding agentic AI capabilities into the products you ship to customers; and (3) AI-Powered Workforce - transforming how your people work internally with AI-augmented operations and productivity. Udaya works closely with customers to bring these pillars together, guiding organizations move from incremental AI adoption to a true AI-First transformation.

High level schedule:

• 4:30 - 4:45: Arrivals
• 4:45 - 5:45: Presentation & Demo
• 5:45 - 6:00: Q&A
• 6:00 - 6:30: Networking

Important instructions

• Event starts at 4:45 pm EST, but please allow at least 15 minutes for security to process registration
• Please ensure that your meetup profile has your full name. Both first and last name are required and we will not be able to register attendees with just abbreviations or incomplete names.
• There is an optional networking and Q&A event at the end of the meetup