03:14 UTC. Site search starts returning 503s. Not slow, not degraded — down. The on-call engineer's first Kibana query for search-service error rates times out too, which is its own bad sign, because Kibana is querying the same cluster that's supposedly the problem.
the setup
The cluster in question is a six-node self-managed Elasticsearch 8.x deployment handling two very different jobs. During the day it serves product and content search for the site, low latency, high query volume, nothing exotic. Overnight it also feeds a batch export: a Node job called order-events-exporter that scrolls the order_events index, about 600 million documents, and streams every doc into Snowflake for the analytics team. It's run at 03:00 UTC for two years without incident.
The export uses the Scroll API, which works by taking a point-in-time snapshot of the index's segments the moment the first request runs, then letting you page through that snapshot with a scroll_id until you're done. The snapshot is what makes it consistent, but keeping it alive costs the cluster something: every open scroll context pins the segments it's reading in place, which blocks merges and keeps otherwise-deletable segments resident in memory until the context is closed or its keep-alive expires.
the scramble
The paged engineer's first theory was a hot shard from a relevance-ranking deploy that had shipped the previous afternoon. It touched the search query builder, so it was the obvious suspect. Rolling it back didn't help; the 503s kept coming at the same rate.
Second theory: traffic spike. The dashboard showed request volume roughly flat against the same time yesterday, which ruled that out inside a few minutes, but the CPU graph in the same panel was doing something odd, climbing in a slow, steady ramp rather than the spiky pattern a traffic surge produces. Steady ramps usually mean something is accumulating, not spiking.
Third theory, and the one that actually got investigated properly: a JVM heap problem. Old-gen usage on the two data nodes handling the order_events shards was sitting above 90%, high enough to explain GC pauses long enough to look like request timeouts. But "the heap is full" isn't a root cause, it's a symptom, and nobody could yet say full of what.
the hunt
The cluster logs had already said what was wrong, twenty minutes before the page fired. It just hadn't been paired with the right alert:
elasticsearch.log, 02:54:11 UTC
[order-events-node-2] TooManyScrollContextsException: Trying to create too many scroll
contexts. Must be less than or equal to: [500]. This limit can be set by changing the
[search.max_open_scroll_context] setting.
That exception is real and specific: Elasticsearch caps concurrent open scroll contexts at 500 by default, cluster-wide, precisely because each one is a standing memory liability. Once the cap is hit, every new scroll request gets rejected, which explained why the export job had been failing since shortly after 02:54, but not yet why site search itself was down; a rejected scroll creation shouldn't touch ordinary, non-scroll search queries at all.
GET _nodes/stats/indices/search gave the number that connected the two:
GET \_nodes/stats/indices/search
"order-events-node-2": {
"search": {
"open_contexts": 1847,
"scroll_current": 1847,
"scroll_total": 2214
}
}
1,847 open contexts, well past the 500 the cluster is supposed to allow. That number only makes sense if contexts were being created faster than they were ever being closed or expired, and closed faster than they were also being renewed past their keep-alive, which pointed straight at the export job's retry logic:
order-events-exporter.js, before
async function exportBatch() {
for (let attempt = 0; attempt < 5; attempt++) {
try {
let res = await client.search({
index: 'order_events',
scroll: '10m',
size: 5000,
body: { query: { match_all: {} } },
});
return await drainScroll(res);
} catch (err) {
log.warn(export attempt \${attempt} failed, retrying\, err);
// falls through to the next loop iteration and calls
// client.search() again from scratch
}
}
throw new Error('export failed after 5 attempts');
}
On a client-side timeout, the catch block didn't resume the existing scroll_id, because it never had one to resume, the request had already timed out client-side before the response carrying it arrived. It just looped and called client.search() again, which creates a brand-new scroll context server-side, every single time. The old context the timed-out request had actually created was never told to close.
Normally this would be forgiving. A scroll context's default keep-alive is short, and an abandoned one expires and gets cleaned up on its own. But eight months earlier, a different slowness ticket had bumped this job's keep-alive from 1m to 10m "to give the export more breathing room," and three weeks before this incident, a mapping change had added a large nested line_items field to order_events to support a new analytics report. Scroll pages that used to return in roughly 80ms were now taking 600-700ms under load, comfortably past the export client's 5-second per-page timeout during the cluster's overnight compaction window. The job was retrying constantly, each retry left a fresh 10-minute-lived context behind, and contexts were piling up roughly ten times faster than they were expiring.
the find
Root cause: the export job's retry loop restarted the scroll from scratch on every timeout instead of resuming or explicitly closing the abandoned context, and a keep-alive that had been widened months earlier for an unrelated fix let each orphaned context live ten times longer than the default. Combined with a recent mapping change that slowed scroll pages enough to trigger the retry loop constantly, contexts accumulated past the 500-context cap within about twelve minutes of the job starting. Past that point, the pinned segments those contexts held open pushed old-gen heap usage over the cluster's parent circuit breaker threshold, which trips for all request types, not just scroll, once triggered:
elasticsearch.log, 03:13 UTC — one minute before the page
[order-events-node-2] [parent] Data too large, data for [] would be
[16.2gb/95%], which is larger than the limit of [15.9gb/94%], real usage: [16.1gb],
new bytes reserved: [112kb]: circuit_breaking_exception
That's the line that took down site search. The parent circuit breaker doesn't distinguish between a batch export's scroll request and a shopper's product query, once it trips it rejects both.
the fix
Immediate mitigation was a manual context flush, which closes every open scroll on the cluster and gives the circuit breaker room to reset:
manual recovery during the incident
curl -X DELETE "localhost:9200/_search/scroll/_all"
Heap usage dropped from 95% to 61% within about ninety seconds and search traffic recovered on its own. The durable fix had two parts. First, the retry logic now resumes by scroll_id and always cleans up on final failure instead of leaking:
order-events-exporter.js, after
async function exportBatch() {
let scrollId;
try {
let res = await client.search({
index: 'order_events',
scroll: '1m',
size: 5000,
body: { query: { match_all: {} } },
});
scrollId = res._scroll_id;
return await drainScroll(res, scrollId);
} finally {
if (scrollId) {
await client.clearScroll({ scroll_id: scrollId }).catch(() => {});
}
}
}
Second, the export was migrated off the Scroll API entirely, onto search_after with a point-in-time (PIT) ID, which is Elasticsearch's own recommended replacement for scroll-based deep pagination and defaults to a much shorter, non-negotiable keep-alive per request rather than one long-lived session an app can forget to close. The keep-alive that had been widened months earlier no longer exists as a setting to accidentally leave too generous.
the aftermath
38 min Site search returning 503s cluster-wide
1,847 Peak open scroll contexts against a default cap of 500
12 min Time from job start to the context cap being exceeded
0 Incidents since migrating off Scroll to search_after + PIT
A new Datadog metric now tracks open_contexts per node with an alert at 300, 60% of the default cap, so the next accumulation gets caught while it's still just a batch job failing, not the whole cluster.
- A retry loop that doesn't know how to resume isn't retrying, it's restarting, and restarting a stateful server-side operation without cleaning up the state it already created is how you turn a transient timeout into a resource leak.
- A keep-alive bump made for one incident months ago can quietly change the blast radius of an unrelated bug later. Nobody revisiting the retry logic knew the keep-alive had been widened, or that it mattered.
- Elasticsearch's circuit breaker protects the whole cluster by design, which is exactly why an isolated batch job's misbehavior became a site-wide outage instead of staying contained to itself.
-
TooManyScrollContextsExceptionwas in the logs twenty minutes before the page fired. The gap wasn't missing information, it was a missing alert on a log line nobody had thought to wire up, because nothing had ever hit that limit before.
The Scroll API had been the right tool for this export for two years. It only stopped being the right tool the moment something downstream, a slower mapping, a wider keep-alive, a retry loop that didn't know its own state, started leaning on the one property Scroll assumes you'll respect: that whoever opens a context is the one responsible for closing it.
