The first version of the pipeline was not the naive one. I had already read enough horror stories to know that calling yt-dlp and ffmpeg inside a Flask request handler would tie up a Gunicorn worker for the length of a download, so the first commit already had a Redis queue and an RQ worker.
That turned out to be the easy part. The queue was never the problem. Everything around it was.
This is what actually broke, or nearly did, in the first month of running TubeMixLab, a browser tool that takes several YouTube links plus start and end times and renders one MP3. Each item is small. Together they are most of the difference between a demo and a service.
The shape of the pipeline
For orientation, the request flow:
browser ──POST /api/resolve──▶ Flask ──yt-dlp (metadata only)──▶ Redis cache (7 days)
browser ──POST /api/mixdown──▶ Flask ──enqueue──▶ Redis queue
│
RQ worker process
├─ yt-dlp: download each source once
├─ ffmpeg: atrim / afade / volume / concat
└─ write job.meta progress to Redis
browser ──GET /api/mixdown/<id>── polls every 700 ms ──▶ reads job.meta
browser ──GET /api/download/<id>─▶ file, link valid 30 minutes
Two Gunicorn web workers, two RQ workers, one Redis. Resolve stays synchronous because it is a metadata fetch that finishes in seconds. Everything that downloads or encodes goes through the queue.
Break 1 (avoided): the timeout that would have leaked processes
This one I got right on day one, so it is here as the thing to copy rather than a war story.
subprocess.run(cmd, timeout=20) looks like it handles a hung download. It kills the process you started. It does not kill the process that process started.
Modern yt-dlp spawns a Node child for its JavaScript challenge solving. Kill the parent on timeout and the Node child keeps running, holding its connection and its memory, with nobody left to reap it. A few dozen of those during an afternoon of throttling and a small worker box is out of RAM for a reason that appears nowhere in your logs.
The fix is to put the whole tree in its own process group and kill the group:
proc = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
start_new_session=True, # new process group, yt-dlp + its children
)
try:
stdout, stderr = proc.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
proc.communicate() # reap
raise
start_new_session=True is one argument. I only knew to add it because I had been bitten by the same thing with a different tool before. Every subprocess in the codebase that can time out uses this pattern.
Break 2: the metadata race at enqueue time
Each job carries some metadata the web process needs later: which tool the export came from, which source ids were used, so the download route can name the file sensibly. The obvious code was:
job = queue.enqueue(run_mixdown_job, clips, bitrate)
job.meta["tool"] = tool
job.meta["sourceIds"] = source_ids
job.save_meta()
Exports started coming back with the generic fallback filename. Not all of them. Enough to notice.
The worker dequeues within milliseconds of enqueue() returning, and the very first thing the job function does is write its own progress into job.meta and call save_meta(). RQ's save_meta() writes the whole hash, not a field. So the sequence was: enqueue, worker saves {stage: downloading}, web process saves {tool, sourceIds} on top of an object it loaded before the worker touched it, or the other way around. Whoever wrote last won, and half the time that was the wrong one.
The fix is to hand the metadata to enqueue() itself so it is written atomically with the job:
job = queue.enqueue(
run_mixdown_job, clips, bitrate,
job_timeout=JOB_TIMEOUT_SECONDS,
meta={"tool": tool, "sourceIds": source_ids},
)
General rule I took from this: if two processes can write the same Redis hash, the write has to happen in one place, and "immediately after enqueue" is not one place.
Break 3: retries that don't understand why they failed
Some fraction of yt-dlp calls fail for reasons that have nothing to do with the video. YouTube's bot check rejects the egress IP. A proxy is slow. A regional exit gets "Video unavailable" for a video that resolved fine a minute earlier from elsewhere. If you retry three times on the same path, you fail three times.
The retry loop ended up doing three things a plain loop doesn't:
-
Classify stderr into a small set of kinds:
bot_check,proxy_error,download_403,not_found,content_error. This is a handful of substring checks against yt-dlp's messages, nothing clever, but it turns "it failed" into something you can count and act on. -
Change the path on retry. Each attempt picks a different egress from a pool, excluding ones already tried in this call, and a
bot_checkresult puts that egress into a cooldown for a while. - Keep the attempt budget inside the job budget. A single source gets up to three attempts with a 600 second ceiling each. A mix with eight sources cannot afford 8 × 3 × 600 seconds, so the per-source attempt count scales down with the number of sources, and the worst case always fits inside the job timeout.
The retry count is not a number you can reason out in advance. Two attempts seemed plenty; live testing against a fresh pool showed back-to-back bad exits often enough that three was measurably better. You have to look at the failure log.
Also: --extractor-retries 0. yt-dlp has its own internal retry loop, and by default it will hammer the same failing path three times before your outer loop gets a chance to rotate. Turn it off and own the retries yourself.
Break 4: progress across two subprocesses
Users wait anywhere from ten seconds to several minutes. A spinner is not acceptable for that range. But the job runs two different tools in sequence, and neither knows about the other.
The split that felt honest: downloads are 0 to 40 percent, encoding is 40 to 99, and 100 is reserved for "the file exists and the link is live".
Download progress is coarse. Each source that finishes downloading advances the bar by its share of the 40. Encoding progress is fine-grained because FFmpeg will report it if you ask:
cmd = ["ffmpeg", "-y", *inputs,
"-filter_complex", filter_complex, "-map", "[out]",
"-c:a", "libmp3lame", "-b:a", f"{bitrate}k",
"-progress", "pipe:1", "-nostats", str(output_path)]
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
for line in proc.stdout:
if line.startswith("out_time_ms="):
out_ms = int(line.split("=")[1])
frac = min(1.0, out_ms / 1000 / total_output_seconds)
set_meta(progress=min(99, 40 + int(frac * 59)))
-progress pipe:1 streams key=value lines to stdout, and out_time_ms is how much output has been encoded so far. Divide by the expected total, which you know because you built the filter graph from the trim ranges, and you have a real percentage. The web process never runs any of this; it only reads job.meta from Redis when the browser polls, so a web worker restart loses nothing.
The filter graph
This is the one place the product logic lives, and it is short. For each clip: trim, reset timestamps, optional fades, volume. Then concatenate.
def build_mixdown_filter(clips):
inputs, parts, labels = [], [], []
for i, c in enumerate(clips):
inputs += ["-i", c["path"]]
length = c["outSec"] - c["inSec"]
fades = ""
if c["fadeInSec"]:
fades += f",afade=t=in:st=0:d={c['fadeInSec']:.3f}"
if c["fadeOutSec"]:
fades += f",afade=t=out:st={length - c['fadeOutSec']:.3f}:d={c['fadeOutSec']:.3f}"
parts.append(
f"[{i}:a]atrim=start={c['inSec']:.3f}:end={c['outSec']:.3f},"
f"asetpts=PTS-STARTPTS{fades},volume={c['volume'] / 100:.3f}[a{i}]"
)
labels.append(f"[a{i}]")
parts.append(f"{''.join(labels)}concat=n={len(clips)}:v=0:a=1[out]")
return inputs, ";".join(parts)
Two details that bit me. asetpts=PTS-STARTPTS after atrim is required, or the trimmed segment keeps its original timestamps and concat produces gaps or overlaps. And the fade-out start time is relative to the trimmed clip, not the source, which is why it is computed from length, not from outSec.
The same source used twice in one mix is downloaded once. Sources are cached on disk keyed by video id, and the resolved metadata (title, duration, thumbnail) is cached in Redis for seven days, so the second person to paste a popular link never waits for yt-dlp at all.
Break 5: metrics that lied without erroring
I had a Prometheus counter for "proxy put into cooldown". One day a Grafana panel running a plain increase() over it reported about 93,000 cooldowns. There had been a handful.
Gunicorn runs two web workers. Each one had its own in-memory copy of every counter, and /metrics returned whichever worker happened to answer the scrape. Prometheus saw the value jump between two unrelated series on alternate scrapes, read each drop as a counter reset, and dutifully added up the phantom increases. No error anywhere. Just a number that was wrong by four orders of magnitude.
The fix for the web tier is the documented one: set PROMETHEUS_MULTIPROC_DIR, build /metrics from a MultiProcessCollector, and add a Gunicorn child_exit hook so a recycled worker's files get cleaned up. One detail that cost an afternoon: gauge files for a dead worker are only removed for the live* gauge modes, so a gauge declared as max leaves a dead worker's last reading stuck in every future aggregation. livemax is what you want.
The RQ workers were a separate problem. Counters incremented inside a worker process live in that process's memory and the web tier never sees them at all. For worker-side events, job failures and per-attempt outcomes, I use a capped Redis list as an event log and a small admin page that reads it. Less elegant than a metric. Actually correct, which matters more.
Smaller things that turned out to matter
- Validate trim ranges before enqueueing. Clamp in and out to the source duration, reject out ≤ in, cap the clip count. A bad range that reaches FFmpeg fails after the downloads, which is the expensive moment to fail.
-
Decouple the download link from the job. RQ's
result_ttlcontrols how long the job record lives. The link lifetime should be a product decision, so the worker writes a small "export ready" record with its own 30-minute expiry and the download route reads that, not the job. - Decide what you won't fetch before you fetch it. Pulling audio from YouTube is a grey area; the tool is only defensible as a general-purpose utility the user is responsible for. So the cheap resolve step runs first, source length is capped, and the site states plainly that users need the rights to what they process. Product decisions, enforced in code before any download starts.
- Rate-limit per route. Polling every 700 ms needs a far higher limit than starting a render.
- Give new job arguments a default. When I added the bitrate option, jobs enqueued by the old web process mid-deploy carried one argument. A worker on new code needs a default or those jobs die.
Where this runs
All of the above is what powers TubeMixLab, which does one narrow thing: paste links, pick start and end times, optionally fade and reorder, get one MP3. The audio cutter on the same site is this pipeline with a single clip, and the plain YouTube to MP3 converter is the same job with no trim at all. None of the code is exotic. Flask, RQ, yt-dlp, FFmpeg, and a lot of small decisions about what happens when each of them fails.
If you are building anything that shells out to long-running tools behind a web app, the two things I would do from the start are process groups on every subprocess and a written answer to "what happens when this step fails" for every step. The queue is table stakes. The failure handling is the product.
Disclosure: I wrote this with help from an AI assistant for structure and editing. Every incident, number and code path is from the real codebase and its commit history, and I checked each one.
