LLM Observability: From Pre-Training to Production
Details
Note & Mandatory: It's mandatory to both RSVP and fill this Google form https://forms.gle/QqPNKqavEGK7phjG9 if either is not done, security won't allow at the venue.
About the event
Traditional monitoring tracks infrastructure health, but it often misses failures unique to LLMs. Models can experience training instability, reward hacking, inference degradation, or hallucinations while conventional dashboards continue to show everything as healthy.
In this session, LLM Observability: From Pre-Training to Production, we'll explore how to monitor, evaluate, and improve AI systems across the entire model lifecycle through practical examples, real-world failure scenarios, and actionable metrics.
### What We'll Cover
- Pre-Training: Gradient norms, loss dynamics, MoE expert balancing, GPU utilization, and training throughput.
- Post-Training: KL divergence, reward dynamics, reward hacking, and model evaluation and regression metrics.
- Inference: Time to First Token (TTFT), Time per Output Token (TPOT), KV cache utilization, GPU memory, batching, and throughput.
- Production Applications: Hallucination detection, groundedness, cost per request, agent and tool-call tracing, task success, and response quality.
### Prerequisites
- Bring your laptop.
- Have OpenCode (https://opencode.ai/) installed in your respective laptop
### Key Takeaway
Learn how to connect training signals, model behavior, inference performance, and application-level outcomes into a unified observability strategy.
A healthy infrastructure dashboard doesn't necessarily mean a healthy AI system.
Join us to go beyond traditional monitoring and understand what it takes to build reliable, efficient, and trustworthy AI systems.
Come curious, bring your questions, and be part of the conversation!