How to evaluate or test the product powered by large language models
Details
Speaker: Narendra Singh Panwar | LinkedIn
Topic: How to Evaluate or Test Products Powered by Large Language Models
Agenda:
🕛 4:00–4:05 PM PDT — Welcome
🕛 4:05–4:55 PM PDT — Narendra Singh Panwar | LinkedIn - How to Evaluate or Test Products Powered by Large Language Models
🕛 4:55–5:00 PM PDT — Q&A & Closing
Summary :
LLMs and AI agents are powering the next generation of products. While the speed of development has increased significantly, evaluating the end-to-end quality of these products remains a challenge.
In this session, Narendra Singh Panwar will share practical learnings and techniques from testing consumer-facing products powered by LLMs and AI agents. The discussion will explore how traditional QA practices need to evolve when testing systems that are non-deterministic and capable of making decisions, generating responses, and completing actions.
The session will cover how teams can evaluate the non-deterministic nature of AI systems, verify factual accuracy, and determine where AI-assisted evaluation can help. Narendra will also discuss the use of LLM-as-a-Judge and where human evaluation is still necessary to assess the quality of AI-powered experiences.
The discussion will also look at AI-powered test automation and continuous customer experience monitoring as ways to support quality evaluation as AI products evolve.
Takeaways :
What attendees will learn:
- How to evaluate the non-deterministic behavior of LLM-powered products
- Approaches for fact-checking and evaluating AI-generated responses
- Where AI and LLM-as-a-Judge can be applied in quality evaluation
- Where human evaluation is still necessary
- How AI-powered test automation can support testing of AI products
- How continuous CX monitoring can help evaluate end-to-end quality
- How traditional QA practices can evolve for products powered by LLMs and AI agents
🎙️ Speaker Opportunities at Our Meetups
Have expertise in QA, leadership, or tech skills you'd love to share? We're always looking for engaging speakers for our monthly meetups. This is a fantastic platform to showcase your knowledge, connect with the community, and inspire others. If you're interested submit your topic here — we’d love to hear from you!