When Should You Actually Worry About a Growing Replication Queue?

Last time, the takeaway was simple: a non-zero replication queue isn't a problem statement, it's a starting point for one. It just means a replica has pending work - fetches, merges, mutations - that Keeper says it needs to do.

That still leaves the actual question unanswered though:

When does a queue go from "normal" to "something's actually wrong"?

This is where you stop looking at the queue as a single number and start treating it as a signal with context - trend, task type, retries, and the exception behind it.

Quick Refresher

Before diving in, the short version from last time:

• Every replica has its own local replication queue
• The queue is generated from Keeper's replication log
• system.replicas gives you the summary (queue_size, absolute_delay, etc.)
• system.replication_queue gives you the detail (task type, retries, last_exception)

With that in place, here's how to actually reason about a growing queue.

Rule #1: A Snapshot Means Almost Nothing

If you check queue_size once and see a number, you've learned almost nothing about whether the replica is healthy.

queue_size = 20

could mean:

• a large batch of inserts just landed, and the replica is about to chew through all 20 tasks in seconds
• the replica has been stuck on the same 20 tasks for the last hour

Same number. Completely different situations.

The only way to tell the difference is to sample it more than once.

SELECT
    database,
    table,
    replica_name,
    queue_size,
    absolute_delay,
    now() AS checked_at
FROM system.replicas
WHERE table = 'your_table';

Run this a few times, a few minutes apart, and look at the shape:

10:00 → queue_size = 20
10:05 → queue_size = 20
10:10 → queue_size = 20
10:15 → queue_size = 21

That's not "a queue of 20." That's a queue that isn't moving, and is now growing. That's the actual signal worth investigating - not the number itself.

If instead you see:

10:00 → queue_size = 20
10:05 → queue_size = 6
10:10 → queue_size = 0

that's a replica doing exactly what it's supposed to do. Nothing to chase there.

Rule #2: Queue Size and Replication Lag Are Not the Same Metric

It's easy to conflate these two, but they answer different questions.

A replica can have a nonzero queue and low delay - because it's processing fast enough that it never really falls behind in any meaningful sense.

A replica can also have a small queue and still be meaningfully behind, if the pending tasks are old and just haven't moved.

That's why system.replicas also exposes:

SELECT
    replica_name,
    queue_size,
    queue_oldest_time,
    absolute_delay
FROM system.replicas
WHERE table = 'your_table';

queue_oldest_time tells you when the oldest task in the queue was created. If that timestamp is recent, the queue is young - probably fine. If it's from an hour ago and the queue hasn't shrunk, you're looking at a genuinely stuck replica, not just a busy one.

Rule #3: num_tries Tells You If a Task Is Actually Failing

Every entry in system.replication_queue carries a num_tries column.

SELECT
    replica_name,
    type,
    create_time,
    num_tries,
    last_attempt_time,
    last_exception
FROM system.replication_queue
ORDER BY num_tries DESC
LIMIT 10;

A task with num_tries = 0 or 1 just hasn't been picked up yet - could be totally normal, especially right after a burst of inserts.

A task with num_tries = 40 and climbing is a task that keeps failing and retrying. That's a completely different problem than "the replica is just busy." Something is preventing that specific task from completing, and it will keep retrying until whatever's blocking it is resolved.

This is usually the fastest way to tell "backlog" apart from "broken."

Rule #4: last_exception Is the Real Diagnostic

If num_tries tells you that something is failing, last_exception tells you why.

Common categories you'll actually run into:

• Network / connectivity issues - a replica can't reach the source replica to fetch a part
• Disk space - not enough room to write the incoming part
• Missing or corrupted parts - the part a task depends on isn't available where expected
• Too many parts - ClickHouse throttling merges/inserts because part count exceeds configured limits
• Keeper session issues - the replica lost its session with Keeper and can't coordinate properly

None of these show up in queue_size. They only show up when you actually read the exception text.

SELECT
    replica_name,
    type,
    num_tries,
    last_exception
FROM system.replication_queue
WHERE last_exception != ''
ORDER BY num_tries DESC;

If this query returns nothing, your queue is backed up but not actively failing - it's just working through volume. If it returns rows, you now know exactly what's blocking progress, and you can go fix that specific thing instead of guessing.

Rule #5: Don't Restart Blind

It's tempting, when a queue looks bad, to reach straight for:

SYSTEM RESTART REPLICA your_table;

Sometimes that's the right move - for example, if the replica's Keeper session got wedged and a restart lets it re-establish state cleanly.

But if the actual problem is disk space, a missing part on another replica, or a network issue between nodes, restarting doesn't fix any of that. It just resets the local state and lets the same tasks fail again, possibly after masking the original exception in the process.

The order that actually works:

1. Confirm the queue is genuinely stuck (not just busy) - check the trend
2. Look at num_tries to find which tasks are actually failing, not just pending
3. Read last_exception for those tasks
4. Fix the underlying cause
5. Then consider whether a restart or SYSTEM SYNC REPLICA is needed to help it recover

Skipping straight to step 5 is how you end up chasing the same issue again a few hours later.

The Checklist

When a queue looks concerning, this is the order I actually go through:

1. Is it growing, or just non-zero? - sample queue_size more than once
2. How old is the oldest task? - check queue_oldest_time
3. Is absolute_delay moving too? - queue size and lag should usually track loosely; if delay is flat while queue climbs, something's off
4. Are tasks retrying? - check num_tries per task, not just the total count
5. What does last_exception say? - this is almost always where the real answer is
6. Is it one bad task or the whole queue? - a single stuck GET_PART blocking everything behind it looks very different from broad, evenly-spread failures

Six questions, in that order, will get you to the actual root cause faster than staring at a single queue_size number ever will.

Final Takeaway

A replication queue growing isn't inherently bad - replicas fall behind and catch up constantly, especially under bursty write patterns. The number alone was never designed to tell you whether that's fine or not.

What actually tells you is:

Is the queue moving, and if it isn't, what does last_exception say about why?

That's the real question behind "should I worry about this." Everything else - queue_size, absolute_delay, num_tries - is just context that helps you answer it faster.