Model providers now advertise context windows large enough to hold a codebase or a stack of contracts in a single request. It's tempting to read that number as a green light, but it isn't. A model that accepts a million tokens and a system that can serve a million tokens to real, concurrent users within your latency and cost budget are two different engineering problems. Conflating them is how teams end up with a demo that works and a bill or a p99 latency chart that doesn't.
You serve a 1M-token model in production by treating context length as a cost and latency variable you manage on purpose, not a spec you flip on. Let’s dive into the memory math, the latency cliff, and the cost curve for serving a model with a million tokens of context, plus the caching and routing techniques that production teams use today to keep all three in check. Plus, a straight answer on when long context is the wrong tool for the job and retrieval should do the work instead.
Key takeaways:
- Serving a 1M-token context model in production is a memory, latency, and cost problem layered on top of a model capability. Supporting long context and serving it reliably are not the same claim.
- Serving long context effectively means stable agent sessions and predictable spend, rather than retries, timeouts, and a bill that scales with the window instead of the question.
- The real decision isn't whether to "turn on" long context, it's which combination of caching, prefix reuse, and retrieval fits your workload's cost and latency budget.
- Production-grade 1M-token serving combines general techniques, including KV cache economics and prefix-reuse approaches like SGLang's RadixAttention. Use DigitalOcean Inference Router for cache-aware routing, plus retrieval-augmented generation (RAG) as the alternative and complement.
What "serving" a long-context model actually means
Context is everything you send to a model in a single request, measured in tokens: the prompt, the documents, the conversation history. A 128K-token window is roughly a novel's worth of text, and frontier closed models now advertise up to 1M tokens (roughly, ten novels). That number describes what the model can read in isolation, but not what happens to time-to-first-token, batch size, or your GPU bill once dozens of those requests land on the same cluster at once.
Support is a model capability, and performance is a serving question. The two come apart the moment real traffic shows up.
Two mechanics decide whether a long-context request is slow to start or slow to finish:
- Prefill: The compute-bound pass where the model reads your entire input before writing anything.
- KV cache: The memory that stores each token's key and value vectors so the model doesn't reread the input on every step.
Dig in to how prefill/decode disaggregation works: the two-phase split behind every long-context serving decision below.
Why its worth optimizing for long-context
Getting long-context serving right pays off in multiple ways.
Here's what's actually at stake once you're running this in production:
- Stable agent and document workloads: A correctly cached long-context session makes it possible for a coding agent or a multi-turn document review to hold state across dozens of turns without resending the same 100K-token context on every message.
- Costs you can forecast: Once you know whether traffic is cache-friendly or retrieval-shaped, it's easier to accurately predict spend.
- No silent accuracy loss: Long-context serving done well accounts for the fact that models answer less reliably as input grows. This happens by routing precision lookups to retrieval instead of trusting long-context recall, and by testing accuracy at the context lengths you'll actually run rather than assuming a bigger model or window is automatically more reliable.
- Latency you can put in an SLA: Understanding the prefill/decode split lets you set a time-to-first-token target you can actually hit, instead of promising an interactive response time a long-context request can't deliver.
- Room to scale without a rebuild: Architectures built around caching and routing from the start absorb growth in context length and concurrent users without the rewrite that a "just resend everything" approach eventually forces.
What to evaluate before you commit to a long-context approach
Long-context serving isn't a single decision, but rather a set of infrastructure and architecture choices considered together:
- Cache-awareness: Check whether the serving layer reuses the KV cache across requests that share a common prefix, rather than recomputing it each time. The DigitalOcean Inference Router, for example, pins a session to the same model so that a warm cache isn't discarded mid-task when switching to a different, cheaper model.
- Batching under long-context load: Confirm the system maintains healthy batch sizes as the average context length grows, using techniques like chunked prefill and paged memory allocation, rather than letting a single long request starve everything queued behind it.
- Isolation from noisy neighbors: Ask whether long-context and interactive traffic run in separate pools. Dedicated Inference on DigitalOcean exists specifically so a large document-processing job doesn't blow up latency for interactive users on the same infrastructure.
- Pricing that reflects the real cost curve: Look for tiered, per-token pricing that acknowledges long-context requests cost more proportionally, plus a prompt-caching rate that's meaningfully cheaper than a full rewrite.
- Observability into cache health, not just request counts: Make sure you can see KV-cache occupancy, eviction, and preemption counts—not just average latency—since that's what actually explains a p99 problem caused by long-context traffic.
Resend, cache, retrieve, or both: what each costs per session
The cleanest way to understand the cost of each approach is to price a full session rather than a single token.
These examples take a 10-turn agent conversation carrying a stable 128K-token context—a codebase or a document set—where each turn adds a short question and gets a short answer:
| Strategy | Use Case | Mechanism | Cost per 10-turn session (128K context) |
|---|---|---|---|
| Resend the full context every turn | Short, one-off sessions where simplicity matters more than cost | Full context re-sent as input tokens on every turn | $3.93 |
| Cache the context; reuse it | Stable, reused context — a codebase, a document set, a long agent session | Prefill cost paid once; later turns reuse the cached KV state | $0.95 |
| Retrieve a relevant slice (RAG) | Large or changing corpora where each query touches a small fraction of the data | Only the relevant slice enters context; the rest of the corpus stays out | $0.33 |
Figures per DigitalOcean's long-context serving tradeoffs analysis, priced at Claude Sonnet's published rates ($3.00/M input tokens, $3.75/M to write the cache, $0.30/M to read it).
Caching cuts the bill roughly 4x against resending; retrieval cuts it roughly 12x. At thousands of sessions a day, that architecture choice matters more than any provider's per-token rate.
The four tradeoffs that decide whether long-context works in production
A 1M-token context window is a spec, not a guarantee that serving it will work. Memory, latency, throughput, and accuracy each break in their own way as context grows, and "supports 1M tokens" and "serves 1M tokens well" can be very different claims.
Memory: the KV cache doesn't scale the way people expect
For every token a model reads, it stores a key and a value vector so it doesn't have to reprocess the input on each new token. That cache grows in direct proportion to context length: the math works out to roughly 43GB for a single 128K-token request on a 70-billion-parameter model in BF16—close to the model's own memory footprint. Push to a million tokens, and the cache alone exceeds what a single high-end GPU holds, before the model's own weights are even loaded.
Latency: why time-to-first-token is the number that breaks
Attention compares every token against every other token, so compute grows with the square of input length: double the context, quadruple the work. That happens entirely during prefill, before a user sees a single word, which is why time-to-first-token ranges from milliseconds with short context to multiple seconds with long context. It also means one long request can hold a GPU for seconds, quietly degrading p95 and p99 latency for every short, unrelated request queued behind it.
Throughput: the bandwidth ceiling no amount of compute fixes
Generating each output token requires reading the entire KV cache from GPU memory, and the read speed is limited by hardware bandwidth. Double the context, double the cache, and tokens-per-second roughly halves, with a limit that has nothing to do with how much compute is sitting idle. A cluster can appear underutilized on a FLOPS dashboard and still serve long-context requests slowly because the bottleneck has shifted to the memory bus.
Accuracy: why a bigger window doesn't guarantee a better answer
Longer context doesn't reliably mean a better answer. A 2026 measurement study found that accuracy at long context lengths doesn't track model size — a smaller model sometimes outperformed a larger sibling — and that a wrong answer just becomes a retry, which turns an accuracy problem into a latency problem measured in total time-to-correct-answer. Production systems that route by expected accuracy at a given context length, not just by cost, cut that wasted retry time substantially.
How production teams are actually solving the long-context problem today
Successful teams aren't fighting the KV cache, they're making it reusable. SGLang's RadixAttention implements prefix caching alongside continuous batching, so a repeated prompt prefix (i.e. a system prompt, a shared reference document, a stable set of few-shot examples) gets computed once and reused across requests instead of recomputed on every call. That keeps throughput comparatively stable even as context length grows into the millions of tokens, which is exactly the workload where naive serving falls apart first.
You can run this pattern yourself on DigitalOcean GPU Droplets, or skip the setup work with the SGLang 1-Click Marketplace app, which handles the underlying infrastructure for you.
DigitalOcean's Inference Router adds cache-aware routing on top of that. It keeps a session pinned to the same model, so a warm KV cache doesn't get discarded mid-task by a switch to a different model chosen for cost reasons.
If you're self-hosting long-context serving on GPU Droplets rather than going through a managed router, three additional levers matter:
- Chunked prefill, which breaks a large prefill pass into pieces so it doesn't stall the rest of the batch.
- Tuning max batch size and max batched-token limits for your actual mix of context lengths.
- Keeping long-context traffic in a separate pool from interactive traffic so one doesn't degrade the other.
When time-to-first-token degrades as context grows, DigitalOcean's inference-debugging guidance points to KV-cache pressure—not the scheduler—as the usual culprit. This is why watching KV-cache occupancy, eviction, and preemption counts diagnoses the problem faster than staring at a scheduler queue.
Do you even need long context? Choosing between caching, retrieval, or using both
If a model accepts a million tokens, it's fair to ask why you'd also use a vector database, a chunking pipeline, and a retrieval step. The answer is that the price of skipping retrieval includes the quadratic prefill cost, the multi-second time-to-first-token, and the tiered per-token pricing. They’re all paid on every request—even when the answer only needed to use three paragraphs of the input.
As a practical rule, long context plus caching wins when your working set is stable and reused. For example, many requests against the same document set, or a day-long agent session where the context is the state. RAG wins when your corpus is larger than any context window, changes frequently, or is mostly irrelevant to any single query. Precision lookups favor RAG even when the full corpus would technically fit, since retrieval keeps the model working at the shorter lengths where it's most reliable. Most production systems end up using both: retrieval to decide what enters the context, a long window to hold a generous amount of it, and caching wherever it repeats.
Long-context LLM serving FAQ
How do you serve long-context (1M token) models in production?
Serving a 1M-token model in production means treating context length as a cost and latency variable you manage on purpose, not a spec you flip on, since supporting long context and serving it reliably are not the same claim. It requires managing KV cache memory, the quadratic prefill cost, and the resulting time-to-first-token, then choosing the right mix of caching, prefix reuse, and retrieval for your workload. The DigitalOcean Inference Router applies cache-aware routing to this problem by pinning a session to the same model so a warm KV cache isn't discarded mid-task.
What is disaggregated prefill and decode in LLM inference?
Prefill is the compute-bound pass where the model reads your entire input before writing anything, while decode is the token-by-token generation that follows. Attention compares every token against every other token during prefill, so compute grows with the square of input length, meaning one long request can hold a GPU for seconds and degrade latency for every short request queued behind it. Chunked prefill breaks a large prefill pass into pieces so it doesn't stall the rest of the batch.
What metrics matter for LLM inference observability (TTFT, TPOT, ITL)?
Time-to-first-token is the number that breaks under long context, ranging from milliseconds with short context to multiple seconds with long context, because it's entirely determined by the prefill pass. Beyond latency averages, observability into KV-cache occupancy, eviction, and preemption counts is what actually explains a p99 problem caused by long-context traffic, not request counts alone. KV-cache pressure, not the scheduler, is the usual culprit when time-to-first-token degrades as context grows.
Which inference provider has the best caching to reduce token costs for long autonomous coding sessions?
Caching the context and reusing it lets a coding agent hold state across dozens of turns without resending the same 100K-token context every message. SGLang's RadixAttention implements this as prefix caching alongside continuous batching, computing a repeated prompt prefix once and reusing it across requests. DigitalOcean supports this pattern on GPU Droplets or through the SGLang 1-Click Marketplace app, and the Inference Router adds cache-aware routing so a warm cache isn't discarded by a mid-task model switch.
How do you model cost-per-token for LLM inference?
The cleanest way to model cost is pricing a full session rather than a single token: resending the full context every turn costs $3.93 for a 10-turn, 128K-context session, caching and reusing that context brings it to $0.95, and retrieving only the relevant slice (RAG) brings it to $0.33. Caching cuts the bill roughly 4x against resending, while retrieval cuts it roughly 12x, a difference that matters more at scale than any provider's per-token rate. DigitalOcean prices around this same curve, with tiered per-token rates that account for long-context requests costing more than proportionally, plus a prompt-caching rate meaningfully cheaper than a full rewrite.
Serve long-context models without managing the cache yourself
The DigitalOcean Inference Engine prices per token by model, with prompt-caching rates listed right next to the standard rate. That means you can work out what a long-context session costs before you run it, not after the invoice shows up. The Inference Router automatically applies cache-aware routing, pinning a session to the model whose KV cache is already warm rather than discarding it during a mid-task switch.
That gets you:
- Automatic model-to-request matching, so cost- or latency-sensitive requests route to the right model without the need for writing the routing logic yourself.
- Dedicated Inference to isolate long-context, latency-sensitive workloads from noisy-neighbor traffic.
- Serverless-to-dedicated scaling with no migration step, so a workload that outgrows shared capacity doesn't require re-architecting.
Get started with DigitalOcean's Inference Engine →
Hippocratic AI cut prefill latency 2x on long-context clinical sessions running production inference on DigitalOcean's cache-aware routing.
https://www.youtube.com/watch?v=WeAAISnnNVA
These customer-reported results reflect Hippocratic AI’s methodology, workload, configuration, and measurement period and may not be representative of results in other environments. Results in customer environments may vary depending on configuration, implementation, and usage. Results and/or savings are not guaranteed.
Any references to third-party companies, trademarks, or logos in this document are for informational purposes only and do not imply any affiliation with, sponsorship by, or endorsement of those third parties.
