The assumption has been simple: if an agent is going to work for a long time, give it a long memory. Every tool call, every file, every dead end stays in the prompt, because throwing it away might cost an answer. AlphaSignal’s Sunday deep dive, written by Ben Dickson, reports a Nvidia study that tested the opposite. Across long-horizon and long-input tasks, models could often work from a much shorter context and keep their accuracy.
The cost of the long memory is not abstract. When an LLM writes a token it must attend to everything that came before it. The researchers measured three open-weight models on NVIDIA H100 GPUs. Cutting the configured context window from 256,000 tokens to 16,000 increased output-token throughput by 3.6 times for GLM 5.2, 5.6 times for Kimi K2.6, and seven times for Qwen 3 Coder Next.
On the work itself they compared a baseline that kept the full conversation with a compaction that triggered at a threshold, for example 16,000 tokens. The agent then started a fresh conversation with the original instructions and a shortened state. On SWE-bench Verified, Claude Opus 4.5 solved 70 per cent of tasks without compaction and 74.8 per cent at its best compaction setting. GPT-5.6 Sol improved from 84.4 per cent to 88.8 per cent. For long documents, storing the files outside the prompt and letting the agent search them matched or improved accuracy across almost all of seven benchmarks. In many of those tasks the agent never compacted at all. The gain was in deciding what to read.
A good compaction, the researchers said, is a briefing that preserves only “the gist of the agentic trace.”
A recap of prominent events is the wrong briefing. The instructions that worked kept the original objective, the established facts, the relevant files and results, the completed changes, the failed approaches, the remaining work, and the immediate next action. A more aggressive test reset the conversation after every action and still nearly matched the full-history baseline on SWE-bench Verified, with a median peak context of just 7,000 to 9,000 tokens.
The letter is careful, and so should we be. Models do not all take the same cut. Qwen 3 Coder Next fell from 53 per cent accuracy on SWE-bench Verified to 3.6 per cent under a 16,000-token limit; the letter says the collapse came from verbose outputs that left no room to work. Prompt caching can also make a shorter request more expensive: GPT-5.6 Sol’s cost under standard per-token pricing rose from $167.68 without compaction to $203.12 at the 16,000-token limit. The practical rule is not “always forget.” It is: keep a structured handoff, leave the logs and originals where they can be retrieved, and measure cache hits, tokens, latency and cost per finished task before you tighten the window.