Testing AI Agents: Context, Evals, and Observability 101 for Testers | Chicago
Details
Testing AI Agents: Context, Evals, and Observability 101 for Software Testers
Traditional software testing assumes determinism.
Given the same input, you expect the same output.
AI agents don’t work that way.
When systems become non-deterministic, context-aware, and probabilistic, QA must evolve. Manual prompt checking and “vibe-based validation” are not enough. Testing AI requires instrumentation, evaluation strategies, and observability at a level most teams haven’t yet formalized.
At the 4th Chicago Meetup, we move from theory to engineering discipline-exploring what a professionalized testing lifecycle looks like for AI agents.
Event Details
- Date: 18th March 2026
- Time: 6:00 PM – 8:00 PM CST
- Venue: Association Forum, Chicago, Illinois
Session: Testing AI Agents - Context, Evals, and Observability 101 for Software Testers
Speaker: Michael Kessler – Technology Leader / Business Developer
About the Session
In this technical session, we shift from manual experimentation to structured verification.
Michael Kessler will demonstrate a live, pre-built AI agent constructed with LangChain, then walk through how to instrument it for reliability in real-world environments.
This session focuses on:
- Designing evaluation frameworks for probabilistic systems
- Instrumenting agents for observability and traceability
- Treating orchestration logic as a testable surface
- Moving beyond intuition toward measurable quality
You’ll gain a practical understanding of how to apply structured QA practices to AI-driven systems-bridging the gap between experimentation and production readiness.
Key Takeaways
- Moving from “Vibes” to Verification – replacing intuition with measurable evaluation strategies
- Debugging the “Black Box” with Observability – understanding agent reasoning and trace flows
- The “Unit Test” for Intelligence – designing meaningful evals for non-deterministic behavior
- Orchestration as a Test Surface – testing how agents coordinate tools, memory, and context
About the Speaker
Michael Kessler is a seasoned technology executive and entrepreneur with over 25 years of experience in the tech industry.
Throughout his career, he has led technology initiatives at organizations including Lucent Technologies, FactSet, and Vivid Seats, and has also built businesses in the fitness and dining sectors.
A specialist in Quality Assurance and Technology modernization, Mike is passionate about evolving traditional testing mindsets to address the challenges of AI-driven systems. He holds a degree in Computer Information Systems from DeVry University and has completed graduate-level coursework at DePaul University and MIT xPRO, focusing specifically on designing and building AI products.
Why You Should Attend
- Learn how to test non-deterministic AI systems with engineering rigor
- Understand evaluation strategies beyond prompt tinkering
- Gain insight into observability patterns for AI agents
- See a live demonstration of agent instrumentation in practice
- Connect with Chicago’s AI and QA community
- Be part of The Test Tribe’s 4th Chicago Meetup
About The Test Tribe
The Test Tribe is the world’s largest software testing community, empowering testers globally since 2018. With 700+ events and 150K+ members across 130+ countries, we bring testers together to learn, grow, and connect through meetups, conferences, cohorts, masterclasses, and more.
By RSVPing, you agree to our Terms and Conditions and Privacy Policy and consent to be contacted by The Test Tribe and our event partners