LLM Observability: From Pre-Training to Production
Details
Note & Mandatory: It's mandatory to both RSVP and fill this Google form: https://forms.gle/QqPNKqavEGK7phjG9 -if either is not done, security won't allow at the venue.
## About the Event
Traditional monitoring focuses on infrastructure health, but monitoring LLM-based systems requires looking beyond CPU, memory, and GPU metrics. Training instability, reward hacking, inference delays, and application-level performance issues can go unnoticed by conventional monitoring tools.
In this event, LLM Observability: From Pre-Training to Production, we'll explore key metrics and monitoring practices across the LLM lifecycle through technical sessions, practical examples, and discussions.
The event will also feature an afternoon hackathon where participants can apply their learnings and present their work.
## What We'll Cover
- Pre-Training: Gradient norms, loss dynamics, MoE expert balancing, GPU utilization, and training throughput.
- Post-Training: KL divergence, reward dynamics, reward hacking, and model evaluation metrics.
- Inference: Time to First Token (TTFT), Time per Output Token (TPOT), KV cache utilization, GPU memory, batching, and throughput.
- Production: Cost per request, response quality, agent and tool-call tracing, task success, and application-level monitoring.
## Event Schedule
9:00 AM – 10:10 AM: Session 1
10:10 AM – 10:50 AM: Session 2
10:50 AM – 11:05 AM: Tea Break
11:05 AM – 11:45 AM: Session 3
11:45 AM – 12:30 PM: Session 4
12:30 PM – 1:00 PM: Q&A and Discussion
1:00 PM – 2:00 PM: Lunch
2:00 PM – 4:15 PM: Hands-on Hackathon
4:15 PM – 5:00 PM: Hackathon Presentations
Lunch will be provided to attendees.
## Prerequisites
- Bring your laptop.
- Install OpenCode before attending: https://opencode.ai/
## Event Sponsor
A special thanks to Grafana and Friends Hyderabad for sponsoring and supporting this event. We appreciate their contribution and partnership in bringing the AI and observability communities together.
## Key Takeaway
Understand the metrics and practices involved in monitoring LLMs across training, inference, and production.
A healthy infrastructure dashboard doesn't necessarily mean a healthy AI system.
Join us for a day of technical sessions, discussions, and hands-on building.
Come curious, bring your laptop, and be part of the conversation!
