From Agent Traces to Scorecards: Review Loops for Production LLM Apps

From Agent Traces to Scorecards: Review Loops for Production LLM Apps

Most teams can get an AI demo working. The harder part is knowing when a prompt, model, or agent change is actually safe to ship.

This session goes past the demo and into the review loop that separates a weekend project from a production LLM app: how teams read agent traces, turn messy behavior into scorecards, calibrate their evals so the numbers actually mean something, and decide where a human still needs to be in the loop. This workshop will contain a hands-on demo, not just slides; you'll see the workflow run, not just hear about it.

If you're building anything with LLMs or agents and you've asked yourself if your demo is really trustworthy in front of actual users, this workshop is for you.

We'll walk through:

• Reading agent traces: what to actually look for when you inspect what a model or agent did, step by step.
• Building scorecards: turning fuzzy notions of "good output" into concrete, repeatable criteria you can measure a change against.
• Calibrating evals: making sure your automated evaluations agree with human judgement often enough to be trusted, and knowing when they don't.
• Deciding where human review still matters: drawing the line between what you can safely automate and what still needs a person, and why that line can move.
• Making the release decision: how teams use all of the above to answer the real question: is this prompt/model/agent change safe to ship?

Who should attend:

• Builders and engineers shipping LLM features or agents who want a reliable way to evaluate changes before release.
• Product and technical leads who have to sign off on the launch decision.
• Data scientists and ML practitioners interested in eval methodology, LLM-as-judge, and its failure modes.
• AI-curious professionals who want to see what "doing this right" looks like beyond the demo.