Beyond Vibe Checks: A Hands On Workshop on Evaluating LLM Apps, RAG and Agents
Details
Note
🚨Important Registration Information🚨
This is an invite-only event with limited capacity.
To be considered for attendance, please register using the registration link provided in this Meetup description. Submitting an RSVP on Meetup alone will not be considered a valid registration.
All registrations will go through a screening process.
If your registration is approved, you will receive a confirmation email.
Link for registration- https://sahaj.ai/events/beyond-vibe-checks-a-hands-on-workshop-on-evaluating-llm-apps-rag-and-agents/
About the Workshop
New models are released every few weeks, each claiming to be better, faster and cheaper. Public benchmarks and leaderboards cannot tell you whether a new model, prompt or retrieval strategy will improve your application or quietly break it. Most teams still answer that question by manually reviewing a handful of outputs.
This hands-on workshop replaces these spot checks with a repeatable evaluation practice. We start with a simple framework that applies to any evaluation: a dataset, an evaluator, the task and how you interpret the results. We then build evaluations for real-world LLM applications step by step, moving from a single prompt to a full RAG pipeline and, finally, a tool-calling agent.
Along the way, we explore how to design reliable LLM-as-a-judge evaluators, account for evaluator noise and integrate evaluation into production through backtesting and online monitoring. We will use LangSmith, but the concepts and practices apply across different tools and frameworks.
What We Will Cover
- Evaluation basics: Datasets, evaluators, tasks and metrics, and the role of public benchmarks such as HumanEval and Chatbot Arena.
- Building datasets: Creating hand-written ‘golden’ datasets, converting production logs into test cases, versioning and splitting datasets by version or category.
- Choosing evaluators: Code checks and unit tests, LLM-as-a-judge evaluations with and without reference answers, custom scoring rubrics and pairwise comparisons for open-ended outputs.
- Evaluating RAG: Answer correctness, hallucination detection (checking whether answers are grounded in retrieved documents), retrieval relevance and evaluating intermediate steps directly from traces.
- Evaluating agents: Assessing final answers, individual tool choices and the complete sequence of tool calls, including identifying hallucinated tool calls and unnecessary steps.
- Comparing versions: A/B testing, regression testing against a baseline, assessing whether a model upgrade is safe in terms of quality, latency and cost, and using repeated evaluations to distinguish genuine improvements from noise.
- Evaluations in production: Backtesting new versions against real traffic, using online evaluators to detect hallucinations and implementing input guardrails for PII and toxicity.
You will leave with a practical evaluation playbook that you can use the next time you change a model, prompt or retrieval setup. You will also get working notebooks for RAG and agent evaluation that you can adapt to your own projects.
Who Is It For?
Engineers building LLM-powered products who are comfortable with Python and want an evidence-based approach to iterating and shipping their applications.
Agenda
09:30 AM – 10:00 AM – Attendance
10:00 AM – 04:00 PM – Workshop
01:00 PM- 02:00 PM – Lunch and Networking
About the Speaker
Suman Paul Choudhury
Solution Consultant, Sahaj Software
Suman Paul Choudhury is a Data Science Consultant at Sahaj with over 10+ years of industry experience in AI and Machine Learning. His work spans Generative AI, Computer Vision, AdTech, customer analytics and large scale ML systems.Suman has extensive experience working with technologies such as PySpark, deep learning, graph based machine learning, LLMs, LangGraph and modern MLOps ecosystems.He holds a Master’s degree in Signal Processing, has contributed to IEEE research publications and has patent involving vision assisted checkout systems.Suman is particularly passionate about translating complex AI concepts into practical systems that solve real business problems and about helping teams move from AI experimentation to scalable, production ready solutions.
🚨Important Registration Information🚨
This is an invite-only event with limited capacity.
To be considered for attendance, please register using the registration link provided in this Meetup description. Submitting an RSVP on Meetup alone will not be considered a valid registration.
All registrations will go through a screening process.
If your registration is approved, you will receive a confirmation email.
Link for registration- https://sahaj.ai/events/beyond-vibe-checks-a-hands-on-workshop-on-evaluating-llm-apps-rag-and-agents/
