Context Engineering

The name is newer than the problem. Prompt engineering assumed a short, hand-written string and a single turn. That assumption died the moment systems started retrieving documents, calling tools, and running for fifty turns. Anthropic's working definition is the useful one: context engineering is "the set of strategies for curating and maintaining the optimal set of tokens during LLM inference" — all the tokens, including everything that lands in the window without a human typing it. The prompt is now a small minority of the context, and it is not the part that fails.

The failure mode is specific and it is not "the model is dumb". It is that a system with a correct prompt, a correct retriever, and a 200K window returns a worse answer than the same system with a third of the input. That is not a bug in any one component. It is a budgeting failure, and the research on it is unambiguous.

Treat the context window like disk and you will fill it. Treat it like an attention budget and you will start making the right calls. The mechanical reason is the transformer itself: attention lets every token attend to every other token, which is n² pairwise relationships for n tokens. Double the input and you have quadrupled the relationships the model must resolve with the same trained capacity. Attention gets spread thinner. Nothing errors — the answer just gets worse.

There is a second, less-discussed reason: training distribution. Shorter sequences are far more common in training data than long ones, so models have proportionally less experience resolving long-range dependencies than the advertised window implies. The result, in Anthropic's framing, is "a performance gradient rather than a hard cliff" — precision decays smoothly, which is exactly why it escapes notice. A cliff shows up in your error rate. A gradient shows up as a system that is quietly mediocre.

1. Position effects are real, large, and survive bigger windows. Liu et al.'s Lost in the Middle (TACL 2024) placed the answer-bearing document at every position in a multi-document QA context and measured accuracy. Performance follows a U-shape: highest when the relevant information is at the very beginning or the very end, materially degraded when it sits in the middle. The result that should stop you cold is the comparison against a closed-book baseline. GPT-3.5-Turbo scored 56.1% with no documents at all and 88.3% with only the correct document (oracle). But in the 20- and 30-document settings, worst-case positioning drove accuracy below the 56.1% closed-book number. Retrieval made the system worse than not retrieving. The paper is explicit: "in the worst case, performance in 20- and 30-document settings is lower than performance without any input documents."

Two follow-on details matter for design. Extended-context models were not exempt — the degradation tracked the model, not the window size. And on a synthetic key-value retrieval task, model families diverged sharply: Claude-1.3 was near-perfect across 75, 140 and 300 key-value pairs, while GPT-3.5-Turbo and MPT-30B-Instruct showed pronounced U-curves. Position robustness is a per-model property you must measure, not a capability you can assume.

2. Reasoning degrades far below the advertised limit. Levy, Jacoby and Goldberg (ACL 2024) built FLenQA specifically to isolate input length: the same reasoning task, padded to roughly 250, 500, 1000, 2000 and 3000 tokens. Across GPT-4, GPT-3.5-Turbo, Gemini-Pro, Mistral 70B and Mixtral 8x7B, average accuracy fell from about 0.92 to 0.68 as input grew to 3000 tokens — degradation visible from around 500 tokens, and their conclusion is that models "quickly degrade in their reasoning capabilities, even on input length of 3000 tokens, which is much shorter than their technical maximum." Three thousand tokens. Against a million-token window. The paper also found that next-word-prediction quality correlates negatively with reasoning performance on their set — the standard metric points the wrong way.

3. Effective context is roughly a quarter of claimed context. Two benchmarks converge here. RULER evaluated 17 long-context models across 13 tasks and found that although models claim 32K or more, only about half maintain satisfactory performance at 32K — despite near-perfect scores on the basic needle-in-a-haystack test everyone quotes. NoLiMa went further and removed the crutch: it builds needles that share minimal lexical overlap with the question, so the model must infer a latent association rather than pattern-match a string. Of 13 models claiming at least 128K support, 11 dropped below 50% of their own short-context baseline at just 32K. Even GPT-4o, one of the best performers, fell from a 99.3% baseline to 69.7%. The mechanism the authors identify is precisely the one above: attention struggles to retrieve without a literal match to latch onto, and that struggle compounds with length.

This is the practical takeaway: needle-in-a-haystack is close to a marketing benchmark. It tests verbatim string retrieval — the easiest possible long-context task. Passing it says almost nothing about whether a model can reason over the same context. Do not accept it as evidence of long-context capability, and do not build your internal eval that way.

4. Degradation is non-uniform and partly counter-intuitive. Chroma's Context Rot report ran 18 models — Claude Opus 4 / Sonnet 4 / 3.7 / 3.5 / Haiku 3.5, o3, GPT-4.1 (plus mini and nano), GPT-4o, GPT-4 Turbo, GPT-3.5 Turbo, Gemini 2.5 Pro / Flash, Gemini 2.0 Flash, and Qwen3 at 235B / 32B / 8B — through controlled variations of input length. The headline: models do not use their context uniformly, and reliability decays with length even on trivial tasks. The specifics are more useful than the headline:

Two footnotes worth knowing before you cite this study: on LongMemEval, Claude Opus 4 refused 2.89% of tasks and GPT-4.1 refused 2.55%; GPT-3.5 Turbo was excluded from the repeated-words task after refusing 60.29% of it on content filtering. Refusal rates are a real confound in long-context evaluation and most write-ups omit them.

Note the tension the table exposes. The two cacheable claimants are the two that are cheap to keep, and the two volatile ones are expensive on every single call. The instinct to trim the system prompt for cost reasons is usually backwards: at cache-read rates it is close to free. The retrieved chunk you added "just in case" is not.

Given a U-shaped attention profile, position is a lever you are already pulling — the only question is whether you are pulling it deliberately. The rules fall out of the research directly:

History is the claimant that grows on its own, so it is the one that needs a policy. There are three distinct mechanisms and they are routinely conflated. They are not interchangeable.

The ordering that works: externalise first, clear second, summarise last. A note on disk costs nothing against the budget and loses nothing. Clearing is cheap and honest — it removes content the model has already metabolised. Summarising is the lossiest option and should be the last resort, because a summariser is an additional model call with its own failure modes, and every compaction cycle is a generation loss. Compact a conversation five times and you are reasoning over a summary of a summary of a summary.

Sub-agents are the fourth option and belong in the same family: a sub-agent explores with its own clean window and returns a condensed 1,000–2,000 token summary instead of its full trace. That is compaction with the compression pushed into a separate context. It works — but see the caveat in section 08 before you reach for it.

Context engineering is usually taught as a quality problem. It is equally a cost problem, and the two have a conflict at the centre that nobody warns you about.

Prompt caching is a prefix match: providers hash the rendered prompt up to a breakpoint, and any byte change anywhere in that prefix invalidates everything after it. The economics are steep enough to dominate design. Cache reads cost roughly 0.1× base input price. Cache writes cost 1.25× for a 5-minute TTL and 2× for a 1-hour TTL. Break-even on the 5-minute cache arrives at just two requests (1.25 + 0.1 = 1.35 vs 2.0 uncached); the 1-hour TTL needs three, but survives gaps in bursty traffic.

The corollary for architecture: put stable content first and volatile content last, always. Frozen system prompt, deterministically-ordered tools, then retrieved context, then history, then the question. A timestamp in the system prompt — datetime.now() in an f-string, a per-request UUID, an unsorted json.dumps — silently costs you the entire cache on every call, and it does so without an error, a warning, or a log line. The way you find it is by checking whether cache-read tokens are non-zero across repeated calls. If nobody is watching that number, nobody knows.

The honest case for a bigger window is narrow and worth stating: a genuinely irreducible input that cannot be chunked without destroying meaning — a whole codebase for a cross-cutting refactor, a single long legal instrument where any clause may condition any other. There, the alternative is not "smaller context", it is "no answer". Take the degradation knowingly and build the eval to measure it. That is a different act from filling a window because it was there.

Context problems are invisible by construction. Nothing throws. The system returns a plausible, slightly worse answer, and the gradient is smooth enough that no alert fires. If you do not measure it, you will not see it — you will just have a product that quietly underperforms its own components.

One closing note on method. Almost every finding on this page is counter-intuitive in a specific direction: less context outperformed more. Shuffled haystacks beat coherent ones. One distractor was enough to hurt. Fifty documents barely beat twenty. Retrieval was sometimes worse than no retrieval. The engineering instinct — when the answer is wrong, give it more to work with — is consistently the wrong instinct here. Budget accordingly.