Microsoft Reactor New York

If you want to move POCs into production, they have to do more than impress. They have to work, at scale. Generative AI demos can feel powerful- fast, fluent, and full of potential. But capability alone doesn’t scale. Without measurement, prototypes stall, trust erodes, and systems never make it to production.

The gap between a compelling demo and a reliable product is rarely the model. It’s the absence of evaluation. To build enterprise-grade AI, you have to measure what you build.

This session introduces the Microsoft.Extensions.AI.Evaluation libraries, designed to make evaluation a first-class part of Gen AI applications. These libraries provide a practical foundation for assessing what matters in real systems: relevance, truthfulness, coherence, completeness, and safety. They include built-in quality, NLP, and safety evaluators, with the flexibility to extend or tailor them to your domain. And as agentic AI takes hold, systems that plan, reason, and take multi-step actions , evaluation becomes even more critical.

We’ll explore how evaluation extends beyond static responses to cover agent workflows, action orchestration, and decision chains. When AI can act, understanding why it acted is as important as the outcome.

By the end, one principle should be clear: You can’t scale AI on intuition alone. You scale it by measuring it.

Key Takeaways:

• Why evaluation is the foundation of LLM Ops, not an afterthought
• How to use Microsoft.Extensions.AI.Evaluation to measure response quality - How to evaluate agentic AI, from workflows to reasoning steps