Why feed millions of tokens if they just eat up the context

There are many misconceptions surrounding language models and their contextual capabilities. Let's break down a few popular myths and see what is actually happening deep inside these technologies.
Myth 1: The more tokens, the better
It seems logical: more tokens mean more information, so the model should perform better. But in reality, as Anthropicnotes, increasing the context can degrade model performance. Once there are too many tokens, the model starts to loop, forget details, and provide vague answers. Why does this happen? Each new interaction requires a review of the entire previous context, which creates quadratic costs for token processing, as warned by vc.ru.
Myth 2: The context window is a bottomless pit
The context window is indeed impressive in size, especially in some models, as mentioned in Chroma with their million tokens. However, this does not guarantee higher quality. The more context there is, the harder it is for the model to focus on key information. As experiments by Chromahave shown, increasing the volume of text does not always lead to improved results, but rather the opposite.
Myth 3: AI agents always know what they are doing
We have all heard stories about how agents can solve tasks faster and more efficiently than humans. However, as Habrnotes, agents often get lost in their multi-step processes. Errors accumulate, and successful execution of complex tasks requires well-thought-out feedback and context management systems. In reality, AI handles only 30% of the work; the rest is tool engineering.
Myth 4: Diffusion models can clean up context
Diffusion models promise to be smarter than autoregressive ones due to their ability to re-evaluate and change unnecessary parts of the context. However, as Habrshows, in practice, the model is more likely to assimilate junk than to clean it up. In theory, it could decide for itself what to keep and what to discard, but this remains more of a hypothesis than a reality for now.
Instead of relying on myths, try evaluating the capabilities of modern AI models yourself at GEMERA AI. Explore how they work through practice.