When models started reading a million tokens at once, everyone declared RAG dead. "Just put everything in the context." Two years later, retrieval is more central than ever. Here's what the hype got wrong.
It's one of the most useful debates in applied AI right now, because getting it wrong wastes money and degrades your product. The instinct that a giant context window replaces retrieval is intuitive — and mostly mistaken.
The seductive argument
Long-context models can accept enormous inputs — entire books, whole codebases. So the reasoning goes: why bother building a retrieval pipeline to fetch the relevant bits when you can just hand the model everything and let it sort it out? Simpler architecture, no vector database, no chunking headaches. On the surface, compelling.
Why "just dump everything" falls apart
Three walls, and you hit all of them fast.
Cost. You pay per token, every single call. Stuffing a massive context into every request is enormously more expensive than retrieving the handful of relevant passages. At any real volume, "put everything in the context" is a budget catastrophe.
Latency. More tokens to process means slower responses. A giant context on every query makes your product sluggish in a way users feel immediately.
Attention degrades. Models don't attend to a huge context evenly — they reliably track the beginning and end and get hazy in the middle. Bury the key fact in the center of a million tokens and the model may effectively miss it. More context can produce worse answers by drowning the signal in noise.
There's also freshness: your knowledge changes, and you can't cram an ever-growing, constantly-updating corpus into a fixed window. Retrieval lets you update a document and have the system reflect it instantly. This is exactly the reasoning behind how I architect AI systems — fetch what's relevant, don't haul everything every time.
They're partners, not rivals
The mature view isn't "RAG vs long context" — it's both, each doing what it's good at. Retrieval narrows a huge, changing knowledge base down to the passages that matter; the long context window then gives the model room to reason over those passages plus the conversation plus the instructions, without you having to be stingy. Bigger windows didn't kill retrieval — they made retrieval more effective by removing the pressure to over-compress what you feed in.
The takeaway
A bigger context window is a better room to work in, not a replacement for deciding what to bring into the room. Retrieval is how you choose; context is where you reason. Anyone telling you long context ends RAG is optimizing for architectural simplicity and ignoring cost, latency, and the way attention actually behaves. The best systems retrieve smartly and then use the generous context well. More on how I build them at www.divyakush.com.
Related reading
- RAG, explained properly: how retrieval keeps an LLM honest — the retrieval half of the partnership, in depth.
