How to test a Test Planning AI Agent

Hi all!

Join our next CTM Online event!

Agenda
7 - 7:10 pm (UTC+2): Intro
7:10 - 8:00 pm (UTC+2): How to test a Test Planning AI Agent (Gil Zilberfeld)
8:00 - 8:15 pm (UTC+2): Q&A - Open Discussion

How to test a Test Planning AI Agent

Let's be honest: building a cool AI agent is pretty easy these days. But keeping it from breaking in production? That's a different story.
We know how to automate stuff. But with LLMs, we need to define what is "working". Then we need to check nondeterministic answers automatically.
We like AI, but we want to make sure it behaves. So, let's do this to a real Test Planning Agent.

In this session, I'm skipping the high-level slides (but keeping the memes) to show you the thinking and tests for our non-deterministic agent.
Here is what we'll walk through:

• Sanity checks and guardrails - the first line of defense.
• Golden datasets - define what's good, bad, and in between.
• Automatic scorecards - translate the quality definitions into automated tests
• Performance and security: A whole set of new things we need to think about in the age of testing AI

And a bit more. Maybe a couple of pirates.
If you're a QA or automation engineer trying to figure out how to apply your testing skills to AI applications, dip your toes, it's a big ocean out there.

Join Zoom Meeting:
https://us06web.zoom.us/j/86812078525
Meeting ID: 868 1207 8525
Passcode: 626792

See you there! 🙂