Study Group: Evals in AI Systems

Join Applied AI Collective for a practical study group on evals in AI systems — how teams measure, test, and improve the quality, reliability, and safety of AI products before and after deployment.
As AI systems move from demos to real-world products, evaluation becomes one of the most important parts of the stack. It is not enough for an AI system to “look good” in a demo. Builders need ways to understand whether the system is accurate, reliable, safe, useful, and improving over time.
This session is designed as a builder-friendly discussion for engineers, founders, researchers, product builders, and AI enthusiasts who want to better understand how evals work in practical AI systems.
What we’ll cover:
• What AI evals are and why they matter
• How evals are used in LLM apps, RAG systems, and AI agents
• Offline evals vs. online evals
• Human evals, automated evals, and LLM-as-judge approaches
• Common failure modes: hallucinations, wrong tool use, bad retrieval, unsafe outputs, and poor user experience
• How evals connect to reliability, guardrails, product quality, and launch readiness
• Open discussion around tools, frameworks, examples, and real-world challenges
Who should attend:
AI builders, software engineers, ML engineers, founders, product managers, researchers, students, and anyone interested in building more reliable AI systems.
No prior expertise is required. Bring your questions, examples, tools you have tried, or AI systems you are currently building.
This is a study group, not a formal lecture. The goal is to learn together, discuss practical approaches, and help each other understand how to evaluate AI systems in the real world.

Let’s build a stronger community around practical, reliable, real-world AI systems.