[PDG 449] Context Rot: How Increasing Input Tokens Impacts LLM Performance

Link to article: https://research.trychroma.com/context-rot
Title: Context Rot: How Increasing Input Tokens Impacts LLM Performance
Content: Large Language Models (LLMs) are commonly assumed to process context uniformly, treating the 10,000th token as reliably as the 100th, but this assumption doesn't hold true in practice. A study evaluating 18 state-of-the-art LLMs, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, found that model performance varies significantly with input length, even on simple tasks. The results show that models become increasingly unreliable as input length grows, demonstrating they do not use their context uniformly.
Slack link: ml-ka.slack.com, channel: #pdg. Please join us -- if you cannot join, please message us here or to mlpaperdiscussiongroupka@gmail.com.

In the Paper Discussion Group (PDG) we discuss recent and fundamental papers in the area of machine learning on a weekly basis. If you are interested, please read the paper beforehand and join us for the discussion. If you have not fully understood the paper, you can still participate – everyone is welcome! You can join the discussion or simply listen in. The discussion is in German or English depending on the participants.