Hands-on AI Evaluation: From Eval Design to Production

An agent may produce the right final answer while choosing the wrong tool, taking 20 unnecessary steps, or making unauthorized calls. If we only look at the final output, the run may appear perfect—but in terms of reliability, efficiency, and permission boundaries, serious problems may still exist.
As AI applications move into production, teams need to understand:

• 🔍 When is an application truly reliable?
• 🧪 Where is it likely to fail?
• 📈 Does each update genuinely improve quality?
• 🛠️ How should failures be diagnosed?
• ⚖️ Can we trust an LLM to evaluate another AI system?

In this practical session, Jessie, a former Senior AI Engineer at Deloitte, will use a real agent execution trace to explore evaluation design, failure diagnosis, LLM-as-a-judge calibration, and production implementation.
You will learn how to build an AI evaluation system that is measurable, diagnosable, and continuously improvable.

## Event Details

• 📅 Date: Monday, 21 September 2026
• 🕒 Time: 7:00–8:00 PM Sydney time
• 💻 Format: Online livestream
• 🎤 Speaker: Jessie — Former Senior AI Engineer at Deloitte
• 💬 Q&A: 10-minute interactive Q&A
• 🎟️ Cost: Free

## Why Does Evaluation Matter? 🔍

Generating a reasonable-looking answer is often only the beginning. In real business workflows, we also need to ask:

• 🔄 Can the application handle different versions of the same task?
• 🧰 Were the tools and parameters used correctly?
• 🧩 Did a component failure affect the overall result?
• 🧠 Can we trust an AI judge’s evaluation?
• 🚀 Do pre-launch tests reflect real user needs?

Evaluation turns these questions into clear test cases, quality criteria, and feedback signals—helping teams compare versions, identify problems, and decide what to improve next.

## What We’ll Cover 🧩

### 1. A “Correct but Failed” Agent Run

Explore a real execution trace where the agent gives the correct answer but still demonstrates tool misuse, unnecessary steps, and unauthorized calls.

### 2. The Core Components of Eval

Understand Dataset, Scorer, and Target, as well as the roles of offline and online evaluation.

### 3. Three Levels of Agent Evaluation

Evaluate agents through:

• ✅ Final results
• 🛤️ Execution trajectories
• ⚙️ Individual components

### 4. LLM-as-a-Judge

Learn how to identify judge bias, build calibration datasets, and validate AI evaluators through human review.

### 5. Build Your First Evaluation Workflow

Start with 10–20 critical cases, integrate evaluation into CI, and continuously add real production failures.

### 6. Who Defines “Good”? 🎯

Understand how domain experts and engineering teams can work together to define standards and maintain evaluation quality.

## What You’ll Take Away 💡

• 📏 A practical framework for measurable AI quality
• 🩺 A structured approach to diagnosing failures
• ⚖️ Strategies for calibrating LLM judges
• 🔧 A practical path to integrating evaluation into CI and production
• 🤝 Clearer collaboration between domain experts and engineering teams

## Who Should Attend? 👥

This session is ideal for:

• 👨‍💻 Engineers and technical leads building AI applications, RAG systems, or agents
• 📦 Product managers responsible for AI product quality and launch readiness
• 🧑‍🔬 Domain experts converting business knowledge into evaluation criteria
• 🧑‍💻 Developers who want to move from an AI demo to a systematic evaluation process

## Q&A and Discussion 💬

Bring your questions about test-case selection, tool-call evaluation, inconsistent judge scores, and how to implement evaluation for an existing AI application.
🌟 Join us online and take the first step toward building more reliable AI applications.
Note: The original event page lists both 7:00–8:00 PM and 9:00–10:00 PM Sydney time. Please confirm the correct time before publishing on Luma.