Why a massive context window doesn't actually stop a neural network from forgetting

Why a massive context window doesn't actually stop a neural network from forgetting

Every second language model release this past year has opened with the same figure: one million tokens, two million, ten million. Marketing presents this as a solved memory problem—the model now 'remembers everything.' As an editor who reads these announcements every day, I'll be blunt: this is a beautiful half-truth. A large window is not the same thing as a good memory. And new research proves this far more convincingly than any advertising slide.

What is a context window, anyway, without the jargon?

A context window is all the text a model sees at once: your question, the chat history, uploaded files, and system instructions. All of this is turned into tokens—the snippets of words the model operates with internally. The context window is the amount of information artificial intelligence can hold in its 'short-term memory' at one time while processing a request and generating a response, and for large language models, it is one of the most important parameters.

An important detail for those writing in Russian: the window is consumed unevenly across languages. The context window is used differently depending on the language: one token in English is usually equal to about 4 characters, while for Russian, due to the specifics of tokenization, a single word can be broken into 3-5 tokens. Therefore, the prompt limit in Cyrillic is reached noticeably faster than the word count would suggest.

If the window is a million tokens, does that mean the model remembers any piece of text equally well?

No, and that is the biggest lie sold by the number on the website. The model is technically capable of accepting a million tokens as input—it won't discard them or throw an error. But accepting and meaningfully using them are two different things. The recent Context Rot Evaluation framework has clearly shown that while the largest vendors are engaged in an aggressive race for context windows, large windows don't work the way we're used to thinking—the model sees the ends of the text perfectly, but the middle 'rots.'

And this isn't just about finding facts. Separate analysis on benchmarks like FLenQA has shown that the degradation concerns reasoning itself, not just memory: even when the task remains equally simple, accuracy drops as the volume of surrounding 'noise' increases.

What is 'context rot' and where did it come from?

The term didn't enter the industry by accident. Developers testing models on real-world tasks—rather than the simplified 'needle in a haystack' test—noticed a strange thing: the more text in the window, the less predictably the model behaves. The Context Rot study shows that performance degrades unpredictably; a token at position 10,000 is processed worse than one at position 100, and as the context grows, the quality begins to noticeably 'jump' even on simple tasks.

The mechanics are simple: every extra token in a prompt is another potential distraction. The model's attention is a limited resource, and when you dump a stack of documents into it, it physically spreads its 'attention' across the entire volume rather than focusing on the necessary fragment.

  • Lost in the middle — information in the middle of a long text is remembered worse than at the beginning and the end.
  • Positional instability — the model finds the same fact differently depending on what surrounds it.
  • Noise growth with length — the longer the input, the lower the signal-to-noise ratio for a specific important piece.

Why the race for window size is largely marketing

Here I have a personal grievance with the industry: the tests boasted about in releases are often outdated and do not reflect reality. An audit of official reports from several labs showed that out of 28 cells in the main vendor result tables, exactly zero contained data from positional stability tests, even though these are the tests that show how a model works with content in the middle of a long text. Simply put: they show one thing in the presentation while hiding the weak spot.

Individual independent measurements provide a sense of the scale of the problem: the NVIDIA RULER benchmark shows that effective context usage is typically only 50-65% of the claimed window volume. This means that the claimed million tokens effectively turn into five or six hundred thousand 'working' tokens in practice, and the rest is ballast that you are paying for anyway.

How, then, should you work with long documents in practice?

The conclusion from all this is not 'don't use long context,' but 'use it consciously.' Here is what actually helps:

  1. Do not upload the entire document 'just in case'—extract only the relevant sections.
  2. Place the most important information at the beginning or end of the prompt—these are the zones the model processes most reliably.
  3. For truly large datasets, use retrieval-augmented generation (RAG) instead of trying to cram everything into a single request.
  4. Break long tasks into stages with a clean context for each, rather than dragging the entire chat history through the whole dialogue.

Should you choose a model based on the number in the window description?

My position as an editor: the context window number is like a car's stated horsepower without mentioning fuel consumption or road quality. You shouldn't look at the ceiling, but at how the model behaves on your specific tasks—long code, a legal document, a series of instructions. Sometimes a model with a smaller window, but more stable behavior in the middle of the text, will prove more practical than one advertising millions of tokens on a banner.

Instead of a conclusion

A context window is not memory in the human sense, but a desk of limited size that is reassembled for every request. The more you pile onto that desk, the higher the risk that an important paper will get lost somewhere in the stack. It is easier to figure out how different models actually behave with long context in practice: in GEMERA AI, you can switch between GPT, Claude, and Gemini in a single chat and compare their performance with large documents on your own tasks—via a Telegram bot or web interface, without complex setup.