Production-Grade Agents: Evaluation, Tracing, Security, and Operations

A successful demonstration does not prove that an agent is ready for production. Production readiness requires measurable quality, end-to-end observability, repeatable testing, controlled releases, and a clear operational model.

In the final session, we will evaluate and operate the agent solution built throughout the series. We will create a representative evaluation dataset covering normal requests, ambiguous cases, tool failures, unauthorized actions, outdated knowledge, and prompt-injection attempts.

Using Microsoft Foundry evaluation and observability capabilities, we will measure task completion, tool-call accuracy, groundedness, relevance, latency, safety, and operational reliability. We will trace requests across the agent, API Management, Logic Apps, Azure Functions, Service Bus, and the worker application using distributed tracing and Application Insights.

During the live demonstration, we will deliberately introduce a faulty tool definition, observe the resulting regression, identify the cause through traces and evaluation results, correct the implementation, and apply a quality gate before releasing the updated version.

The session will conclude with production monitoring, alerting, versioning, rollout and rollback strategies, and agent governance across multiple teams and environments. By the end of the session, attendees will know how to move from “the agent appears to work” to evidence that it is reliable, explainable, supportable, and safe to operate.

Key topics: Microsoft Foundry evaluation Agent and tool-call quality metrics Distributed tracing Application Insights Regression testing safety and adversarial testing CI/CD quality gates Monitoring, incident response, and governance.

This session is a part of a series, learn more here! https://aka.ms/ProBadgeSeries-TY

[eventID:27510]