Monitoring LLM Applications: Traces, Feedback, and Production Quality
Details
This is the 5th workshop in our series to update the LLM Zoomcamp content.
This workshop updates Module 5: Monitoring.
In this hands-on session, Alexey Grigorev will show how to monitor an LLM and RAG application after it is deployed.
You’ll learn how to instrument an LLM pipeline, collect traces and metrics, store chat history, track user feedback, and evaluate answer quality in a running system.
What you’ll learn:
- Why monitoring matters for LLM and RAG applications
- What to monitor in a deployed LLM system
- How to collect traces from an LLM pipeline
- How to instrument LLM calls and retrieval steps
- How to store chat history and user interactions
- How to collect user feedback on generated answers
- How to track answer quality over time
- How to detect issues in retrieval and generation
- How to use dashboards to inspect system behavior
- How to run evaluations on production traces
- How monitoring connects offline evaluation with real-world usage
By the end, you’ll understand how to monitor a deployed RAG application and inspect what happens after users start interacting with it.
Like the other workshops, this will be a live demo with practical tips and time for Q&A.
***
All events in these series:
- Build Your First RAG Application with LLMs
- From RAG to AI Agents: Function Calling and Tool Use
- Vector Databases: Embeddings, Semantic Search, and Hybrid Retrieval
- RAG and Agents Evaluation: Measuring Retrieval and LLM Answer Quality
- Monitoring LLM Applications: Traces, Feedback, and Production Quality
***
## Thinking about Joining LLM Zoomcamp?
This workshop covers the updated content for Module 5 of the LLM Zoomcamp, our free course on building practical LLM applications with RAG, vector search, evaluation, monitoring, and AI agents.
You start with a simple RAG pipeline, then improve it with better retrieval, semantic search, function calling, evaluation, monitoring, and production practices.
The course covers the full lifecycle of an LLM application: from the first working prototype to evaluation, monitoring, and a complete final project.
The new cohort of LLM Zoomcamp starts on June 8, 2026. You can join it by registering here.
##
## About the Speaker
Alexey Grigorev is the Founder of DataTalks.Club and creator of the Zoomcamp series.
Alexey is a software and ML engineer with over 10 years in engineering and 6+ years in machine learning. He has deployed large-scale ML systems at companies like OLX Group and Simplaex, authored several technical books, including Machine Learning Bookcamp, and is a Kaggle Master with a 1st place finish in the NIPS’17 Criteo Challenge.
**Join our Slack: https://datatalks.club/slack.html**