You are Paying to Compute the Same Tokens Twice

This time we have a new talk by Christopher Haar

Every token an LLM generates attends to every token before it. Without a cache, that is quadratic work on every single step, the same keys and values recomputed over and over. The KV cache turns that into linear work. It is the reason inference is affordable at all.

The catch: that cache lives in the GPU memory of one specific pod.
Turn 10 of a conversation. 96000 tokens in the request. Twenty are new. The load balancer picks a different pod. That pod recomputes all 96000 while the cache it needed sits idle one hop away.
Kubernetes treats LLM inference like a stateless REST call. It is not. Every scheduling primitive you trust, round-robin, readiness, HPA, rolling updates, was built to ignore exactly the state that makes inference cheap.
This talk starts with why the KV cache exists, then shows what your cluster does to it. And how vLLM, llm-d, AIBrix and Dynamo fix it: cache-aware routing, prefill/decode disaggregation, KV offloading.