🧪 Evals: One Does Not Simply Ship an Agent

🧪 Evals: One Does Not Simply Ship an Agent

Building an AI agent is easier than ever.
Knowing whether it actually works is much harder.

An agent might handle your initial test cases, but how does it perform when requirements change, edge cases appear, or responses become more open-ended?

Without a structured evaluation framework, it's difficult to measure progress or know whether your changes are actually making the agent better - or how it performs in the first place.

In this hands-on workshop, we'll stand up and explore a working eval framework using a Claude-powered shopping assistant.

Along the way, you'll learn how evaluation tasks are defined, how deterministic and LLM-as-judge graders are implemented, how results are interpreted, and how evals can guide iterative improvements.

You'll leave with a practical reference implementation and a clearer understanding of how modern AI agent evals are built.

What You'll Learn:

• Explain why evals are essential for reliable AI agent development
• Demonstrate a working eval harness using a shopping assistant example
• Teach how to create task definitions, graders, and result summaries
• Show how to improve agent performance in a repeatable way
• Add an advanced LLM-as-judge grader for open-ended evaluation

Hands On Lab: If you'd like to follow along during the workshop, please review the repository setup instructions ahead of time and clone the project with your preferred Claude environment.

Repository: https://github.com/jothsnapraveena/Agentic-Evals

Speaker: Jothsna Praveena Pendyala, Chair of ACM Dallas, is an AI and Data professional focused on agentic AI, evaluation frameworks, RAG systems, and practical GenAI applications. ACM Dallas is a professional computing chapter that supports learning, collaboration, and community engagement through technical events, AI workshops, and professional networking opportunities across the Dallas-Fort Worth area.

Details:
📍 Location: Improving Plano — 5445 Legacy Dr #100, Plano, TX
🕕 Time: Wednesday, July 22 · 6:00–9:00 PM
💻 Livestream Link Here

Agenda:
6:00 – 6:30 PM
Check-in, networking & pizza

6:30 – 7:30 PM
Evals: One Does Not Simply Ship an Agent
Jothsna Praveena Pendyala

7:30 - 9:00 PM
Q&A and Networking

🌐 Stay Connected

If this is your first time joining us, welcome! Join the Prompt Engineers AI community, follow us on LinkedIn, and connect in Slack to stay involved between meetups.
www.promptengineers.ai