We’ve spent a decade giving attention all of our attention. I think that’s exactly backwards.

Every conversation you have with a large language model rests on two pillars.

One is attention — the mechanism the model uses to process information. The other is context — the information the model is actually working with.

They’re easy to confuse, partly because the industry talks about one of them constantly and the other almost never. We obsess over attention. We take context for granted.

That imbalance, I want to argue, is where most AI products quietly go to die.

This is, in a sense, a story about the inhuman side of AI. A human expert carries working memory, a lifetime of consolidated knowledge, and an instinct for what to ignore. A language model has none of these in the way we do. It has a fixed window of text it can see, a brilliant but expensive mechanism for relating the pieces of that text, and no true memory at all once the window scrolls past.

Understanding the relationship between these two pillars is the clearest way to understand both what these systems can do — and where they break.

Two words that get confused

Let’s start with clean definitions, because almost every muddle in this debate comes from blurring them.

Context is the information the model is working with. It’s everything inside the model’s field of view for a single response: your prompt, the system instructions, retrieved documents, prior turns, tool outputs — all of it, expressed as tokens.

Attention is the mechanism the model uses to process that information. It’s the operation, repeated across every layer, that lets each token look at every other token and decide how much each one matters.

A simple way to hold them apart:

Context is the working memory. Attention is the underlying mathematical engine of the Transformer.

Context is what the model thinks about. Attention is how it thinks. One is content; the other is computation.

There’s a second, subtler split worth naming early:

Attention governs the quality of reasoning — how well the model relates ideas and follows a chain of logic. Context dictates the capacity for information — how much it can hold in view at once.

A model can have superb reasoning and a tiny window, or a vast window and shallow reasoning. The two pillars are independent — and they fail independently.

Attention: the mathematical engine

Attention is the idea that gave us the modern era. The 2017 paper that introduced the Transformer was titled, with some swagger, Attention Is All You Need (Vaswani et al.).

Its central move was radical: throw out recurrence and convolution entirely, and build a network whose only real machinery is self-attention — a mechanism where every position in a sequence issues a query and compares it against every other position, producing a weighted blend of the whole sequence for each token.

The payoff is enormous. Because every token can directly attend to every other token, the model captures long-range relationships in a single step instead of passing information hand-to-hand down a chain. It’s also massively parallelizable — which is exactly what let these models scale onto modern GPUs and absorb internet-scale data.

But here’s the part worth being precise about: attention doesn’t give the model knowledge. The knowledge lives in the weights and, in the moment, in the context. What attention provides is relational reasoning — figuring out that the pronoun here refers to a noun forty words back, that this number belongs to that column header. It turns a sequence of symbols into a web of relationships.

And that power has a price written into its arithmetic.

Because each of the n tokens must compare itself against all n tokens, the cost of self-attention grows with the square of the sequence length — quadratic complexity, O(n²). Double the input, and you roughly quadruple the work.

This single fact is the gravitational force behind nearly everything that follows.

Context: the working memory

If attention is the engine, context is the fuel tank — and the dashboard, and the road ahead, all at once.

The context window is the operational boundary of the system: the maximum number of tokens the model can hold in view for a single response. Everything the model “knows” in the moment lives inside that window. Step outside it, and the information simply does not exist.

Crucially, context scales linearly, O(n), in the number of tokens you store. Add a thousand tokens, add a thousand tokens’ worth of storage. No more.

This is the asymmetry at the heart of the whole subject:

The thing you want more of (context) grows linearly. The thing that processes it (attention) grows quadratically.

Context is cheap to hold and expensive to think about.

It also reframes what a “long-context model” really is. When a lab announces a million-token window, it isn’t announcing a million tokens of free thinking. It’s announcing a much larger arena in which the quadratic engine must operate — and a much larger space in which the working memory can degrade. The window is a promise about capacity, not quality. Those are different guarantees, and conflating them is where a lot of disappointment comes from.

“Walk or drive”: the complexity trade-off

Here’s a mental model I find clarifying — call it walk or drive.

For a short trip, walking is fine. For a long trip, you drive. The choice depends entirely on distance.

Attention faces the same decision at every layer. On a 1,000-token prompt, quadratic attention is a rounding error. On a 1,000,000-token prompt, it’s the entire ballgame.

This is why the field has poured energy into linear and sub-quadratic attention — Reformer, Linformer, Performer, and more recently state-space models like Mamba, which achieves linear-time sequence modeling (Gu & Dao, 2024).

But these aren’t free wins. Most sub-quadratic methods trade away something — usually precise recall — to buy their speed, and the gains don’t always hold at very large input sizes.

The honest summary: the quadratic bottleneck has been softened, not solved. The most interesting recent systems are hybrids — they “walk” most of the time and “drive” only when precision demands it.

When the working memory fails

If context were a perfect, uniform store, big windows would be an unambiguous good. They’re not. The model’s working memory degrades in ways that are now well documented — and this is the most practically important part of the whole story.

No true memory. Once a token falls outside the window, it’s gone — not stored, not summarized, not recalled. What looks like memory in a long chat is just the prior conversation being re-fed into the window each turn. The instant it scrolls off, the model is, literally, amnesiac. Everything we build on top — chat history, retrieval, scratchpads — is scaffolding around this absence.

Lost in the middle. Even within the window, not all positions are equal. The influential study Lost in the Middle (Liu et al., 2023) showed models reliably use information at the beginning or end of a long context, but performance sags badly when the relevant fact sits in the middle — even for models marketed as long-context. The window isn’t a flat shelf; it’s a U-shaped valley. This is usually probed with the Needle-in-a-Haystack (NIAH) benchmark: bury a fact at varying depths and see if the model can find it.

Context rot. Most sobering is what Chroma’s 2025 research named context rot. After testing 18 frontier models, the finding was blunt: as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases. Every single model got worse as input grew. A model advertised with a 200K window can show meaningful degradation well before 50K tokens. (Worth noting: this is industry research, not peer-reviewed — but the pattern is consistent across models.)

Context rot is not context overflow. Overflow is hitting the wall. Rot sets in long before you reach it.

The implication for builders is enormous: a 1,000,000-token window does not mean 1,000,000 usable tokens. It means a large room whose far corners are dim. Treating the advertised window as reliable working capacity is one of the most common — and costly — mistakes in production AI.

The hidden wall: why hardware makes it worse

There’s a deeper, physical reason context is hard to scale, sitting below any model architecture: the memory wall.

Over the past two decades, raw compute has grown explosively — roughly 3× every two years in peak FLOPS. But the memory that feeds those chips hasn’t kept pace: bandwidth has grown only ~1.6× and ~1.4× every two years.

In plain terms: the explosive growth in processing power has not been matched by growth in memory — in capacity or speed. The memory wall is the resulting bottleneck — the speed of moving data between memory and the processor. Transformer inference shuttles huge amounts of data (notably the attention key-value cache) back and forth, so the system increasingly spends its time waiting on memory rather than computing.

The bottleneck has migrated. Compute is abundant; context is scarce. Most of our instincts haven’t caught up.

The real costs of long context

Long context isn’t a free lunch you unlock by paying for a bigger window. It carries three structural costs every serious builder must weigh:

  • Security vulnerability. A larger window is a larger attack surface. The more untrusted text you pour in — retrieved docs, web pages, tool outputs — the more room for prompt injection and data exfiltration.
  • The quadratic bottleneck. Longer context doesn’t just cost more to store; it costs disproportionately more to process — in both compute and the memory traffic hardware can least afford.
  • Performance degradation. Lost-in-the-middle and context rot mean quality doesn’t hold steady as the window fills. Beyond a point, more context actively hurts.

The principle that emerges: more context is not strictly better. There’s an optimal amount of relevant information for any task — and overshooting it degrades the very reasoning you were trying to support.

From prompt engineering to context engineering

For a few years, the celebrated craft was prompt engineering: finding the magic phrasing that coaxed the best answer out of a model.

That framing now looks quaint. The natural progression — and the term that has rapidly taken hold — is context engineering: the discipline of curating and maintaining the optimal set of tokens fed to the model during inference.

Andrej Karpathy describes it as “the delicate art and science of filling the context window with just the right information for the next step.” Anthropic formalized it in 2025 as the set of strategies for managing the whole token budget. Prompt engineering is just one small subset of context engineering. The prompt is one ingredient; the context is the entire meal.

The practical toolkit:

  • Structured prompts — put the most important material at the beginning and end, where the model attends best. Fight lost-in-the-middle with placement.
  • Compress and reset — periodically summarize the conversation and discard the raw history. A short, dense summary often beats a long, noisy transcript.
  • Retrieval-Augmented Generation (RAG) — store the corpus externally and retrieve only what’s relevant. RAG is a prosthetic long-term memory bolted onto a model that has none.
  • Linear and state-space architectures — make long context cheaper to process in the first place.

Notice that three of those four aren’t about the model at all. They’re about how a human curates what the model sees.

And the infrastructure is converging here too. Anthropic’s Model Context Protocol (MCP), introduced in late 2024, is an open standard for connecting models to external tools and data. That the industry rallied around a protocol named Model Context Protocol — not “model prompt protocol” — is itself a tell. The frontier has moved to context.

The inhuman side, revisited

There’s a quiet consequence for how we should use these tools.

When a prompt assumes the user is the expert — when the human already knows the answer and is merely seeking confirmation — the model collapses into a validation engine rather than a knowledge-generation engine. It tells you what you wanted to hear, dressed in fluent prose.

The model has no independent foothold in reality, no working memory of its own, no stake in being right. It has only the context you give it and the attention to process it.

Treat it as an oracle and it will flatter you. Treat it as an engine that amplifies the quality of its inputs, and it becomes genuinely powerful. The difference lies almost entirely in context — and in your ability to tell a good answer from a merely plausible one. The tool rewards the prepared mind, which is perhaps the most human thing about it.

So — who wins the AI wars?

Here’s the thesis the whole piece has been building toward.

We’ve spent a decade giving attention all of our attention. It earned it — it’s the engine of the revolution. The mechanism itself is far from finished: reasoning models that “think longer” at inference time, and a steady stream of architectural work, are still pushing it hard. But notice what that frontier is really about — squeezing more out of the computation. It’s mature, well-trodden territory with a lot of capital and talent already pointed at it.

The neglected leverage — the place where comparatively few people are looking — is the other pillar. The frontier of usable intelligence is increasingly about better fuel, better roads, and better navigation. It’s about context.

To be clear, these two pillars aren’t really rivals; the “vs.” is a provocation. Context engineering matters because of how attention behaves over long inputs — they’re two ends of one lever. My argument isn’t that attention stopped mattering. It’s that, relative to its importance, context is where the underinvestment lies.

Attention is getting too much attention. The race is being decided, quietly, on context.

Ask why most ambitious AI products fail in practice. The answer is rarely that the model couldn’t reason. It’s that they mishandled the context — fed it too much, or the wrong things, or stale things, or things buried where it couldn’t see them.

And ask who will win the so-called AI wars. Here I’m making a bet, not citing a result — model capability, data, compute, distribution, and product all matter too. But my bet is that the decisive, underrated differentiator isn’t whoever trains the largest model. It’s whoever handles context the best: who retrieves the right information, prunes the irrelevant, places the critical facts where attention can find them, and respects the limits of a working memory that rots under load.

The two pillars aren’t equals in the way we imagine. Attention is the brilliant, expensive engine we’ve largely figured out how to build. Context is the harder, humbler, more human problem of deciding what to feed it — and it’s the one still wide open.

The winners of the next era will stop asking “how do we make the model think better?” and start asking “what, exactly, are we asking it to think about?”

The model supplies the attention. The hard, decisive, irreducibly human work is supplying the context.

If this resonated, I’d love to hear how you’re thinking about context in your own work — write to me.

References