Building Production-Ready AI Systems: Security, AgentOps & LLM Evaluation

# Building Production-Ready AI Systems: Security, AgentOps & LLM Evaluation

📅 July 23, 2026 | 5:30 PM – 6:30 PM PT
🎥 Virtual Event
🔗 Register on Microsoft Reactor:
https://aka.ms/ProdReady723/m
AI is no longer just about building models.
Today's AI applications require security, evaluation, observability, governance, and continuous improvement to succeed in production.
Join experts from Microsoft and Amazon as they share practical lessons from building, securing, evaluating, and operating AI-powered systems at scale.
As AI applications evolve from prototypes into real-world products, engineering teams face new challenges:
• How do you protect AI systems from prompt injection, tool abuse, and emerging agent threats?
• How do you evaluate whether an LLM is actually performing well in production?
• How do you monitor, debug, and improve AI agents over time?
• How do you fine-tune models for domain-specific workflows and measurable business impact?
This session brings together three critical pillars of modern AI engineering:
🔒 AI Security
⚙️ AgentOps & Observability
📊 LLM Evaluation & Fine-Tuning

***

## Featured Talks

### Securing the AI Stack — From Models to Agents to Infrastructure

Kriti Faujdar
Senior Product Manager, Microsoft Security AI Research
Learn a practical defense-in-depth framework for AI systems, covering prompt injection, jailbreaks, tool and MCP security, memory poisoning, sandboxing, secret management, and infrastructure-level protections.

***

### AgentOps in the Open: Tools for Building, Testing, and Trusting AI Agents

Debjyoti Paul
Applied Scientist, Amazon
Explore the emerging AgentOps ecosystem and learn how teams are tracing agent behavior, evaluating tool calls, monitoring failures, testing prompts and workflows, and building feedback loops for continuous improvement.
Topics include Langfuse, OpenTelemetry, DeepEval, RAGAS, prompt versioning, testing frameworks, and production observability.

***

### LLM-Driven Merge Conflict Resolution

Advitya Gemawat
Machine Learning Engineer, Microsoft
Discover how custom LLM evaluations and Azure OpenAI fine-tuning were used to build an AI-powered merge conflict resolver for one of the world's largest software codebases. Learn practical lessons from deploying LLM-powered developer tools, designing evaluation frameworks, and adapting models to domain-specific workflows.

***

## What You'll Learn

✅ Security best practices for AI agents and applications
✅ How to evaluate LLMs beyond traditional benchmarks
✅ AgentOps tools and observability techniques
✅ Azure OpenAI fine-tuning workflows
✅ Real-world lessons from Microsoft and Amazon
✅ Practical approaches for building trustworthy AI systems

***

## Who Should Attend?

• Software Engineers
• Machine Learning Engineers
• AI Engineers
• Data Scientists
• Platform Engineers
• Product Managers
• Anyone building AI agents, copilots, RAG systems, or LLM-powered applications
Whether you're experimenting with AI agents or deploying production AI systems, you'll leave with practical frameworks, tools, and engineering insights you can apply immediately.
Hosted by PyData Seattle × Microsoft Reactor