I Padded sleep() for 48 Hours. The Event Loop Never Saw the Event

I started with a pytest run that looked healthy and still missed a timeout that production hit daily. Have you ever watched a suite stay green while the same coroutine hung in a real event loop? I had an asyncio worker, a fake upstream, and a test that waited with time.sleep inside an async function. Forty-eight hours later I still believed the flakiness lived in the network, not in the sleep.

Hour 0–8: I blamed the stub

The worker pulled jobs from an asyncio.Queue and called a tiny in-loop stub that was supposed to mimic a slow upstream. When the stub ran slow, the test sometimes passed and sometimes died on wait_for, which I could not reproduce under pdb. Did that mean the stub was racy, or did it mean the test never let the stub run? I added more logging, more sleeps, and a longer pytest timeout, which is the classic way to hide a blocked loop.

What I tried first:

1. Restarted the stub between tests with a fixture that still shared one event loop.
2. Captured timestamps that jumped in two-second slabs after each padded sleep.
3. Raised the client timeout from half a second to five seconds, then to fifteen.
4. Printed asyncio.all_tasks() after the sleep and wondered why the worker stayed pending.

None of that was a measurement, and looking back it was superstition wearing extra timestamps. Could a log line that only flushed after sleep ever tell me the worker was alive? I needed a reproduction that failed the same way every time I blocked the thread.

Hour 8–24: padding sleep() made the suite look calmer

I did what a tired person does and inserted time.sleep of two seconds after enqueueing the job. The test turned green on my laptop, then red in CI, then green again when I reran it on a quieter machine. Why would a longer sleep fix a race unless the sleep itself had become the scheduler? Because time.sleep is a thread-blocking call, and pytest-asyncio was running the test coroutine on the same thread as the event loop.

Here is the lab reconstruction I finally extracted from those forty-eight hours of padded sleeps. Treat this as a local reproduction, not as a production postmortem dressed up with invented metrics. You can paste it into a file and watch the assertion fail without waiting on CI.

# lab_blocked_loop.py
"""Lab reconstruction: time.sleep freezes the thread that runs the loop."""
import asyncio
import time
from dataclasses import dataclass, field


@dataclass
class Worker:
    queue: asyncio.Queue = field(default_factory=asyncio.Queue)
    seen: list[str] = field(default_factory=list)

    async def run(self) -> None:
        while True:
            job = await self.queue.get()
            if job is None:
                self.queue.task_done()
                break
            await asyncio.sleep(0.05)  # stand-in for I/O
            self.seen.append(job)
            self.queue.task_done()


async def broken_test() -> None:
    worker = Worker()
    task = asyncio.create_task(worker.run())
    await worker.queue.put("invoice-1")
    time.sleep(0.3)  # blocks the loop; run() cannot progress
    assert worker.seen == ["invoice-1"]
    await worker.queue.put(None)
    await task

python - <<'PY'
import asyncio
from lab_blocked_loop import broken_test

try:
    asyncio.run(broken_test())
except AssertionError as exc:
    print(type(exc).__name__, exc)
PY

Run it with a plain loop and you get an assertion failure, not a flake, which is a gift. Have you noticed how much easier a deterministic failure is compared with a CI rerun button? If you swap in await asyncio.sleep of 0.3 seconds, the assertion passes, and that swap is how I almost fooled myself twice. asyncio.sleep yields control, time.sleep does not, and that single difference is the whole incident.

Have you checked which sleep your generated test helper actually inserted into the async function? I had not, and the helper name await_ready() sounded innocent enough to survive a skim review. Would you have spotted a sync sleep behind a coroutine-shaped name at eleven at night? I did not, which is why the lab file now forbids time.sleep by grep instead of by memory.

Hour 24–36: an assistant drafted waits, pytest still had to fail

I did not want a cloud pastebin holding fixture dumps, so I iterated wait helpers on MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The assistant drafted Event-based waiters and a couple of asyncio.wait_for wrappers that looked tidy in the editor. I still had to run the reproduction locally, because one draft swallowed CancelledError and another polled worker.seen with time.sleep again. If that free server option is already nearby, keep wait-helper drafts there until pytest fails closed on your machine.

That split is the method: let the assistant propose wait shapes, and keep the proof on a loop you control. Green tests that cannot fail are not tests, and I already had a drawer full of those. Did the generated wrapper raise when the worker never appended, or did it wait until pytest killed the process?

Commands I actually kept:

PYTHONASYNCIODEBUG=1 PYTHONTRACEMALLOC=1 python -m pytest -q tests/test_worker_wait.py
python -c "import asyncio; print(asyncio.__file__)"
rg -n "time\.sleep|os\.system\(['\"]sleep" tests

The debug environment variable does not fix a blocked loop on its own, and I wasted an hour hoping it would. It does make slow callbacks loud once you stop freezing the thread that runs the loop. Would I still enable it after the sleep is gone, or is that just another superstition? I keep PYTHONASYNCIODEBUG enabled now as a seatbelt, not as a diagnosis of the hang.

A wait matrix I wish I had on hour one

The row I now treat as default is timeout plus Event, and anything else needs a written reason in the PR. Have you noticed how many wait helpers are just asyncio.sleep wearing a more professional name? I had three of those helpers, and every one of them passed on a quiet laptop. They all lied under CI load because hope is not a condition the loop can await.

Hour 36–48: the version I would repeat

The repeatable version does three things in order, and I now refuse to skip the third. It starts the worker as a task so the event loop owns the coroutine for the whole wait. It arms an asyncio.Event from the worker after the side effect that the test actually cares about. It waits with asyncio.timeout so a stuck worker becomes a failed test instead of a padded sleep.

# lab_event_wait.py
"""Lab reconstruction: wait on an Event, fail closed with asyncio.timeout."""
import asyncio
from dataclasses import dataclass, field


@dataclass
class Worker:
    queue: asyncio.Queue = field(default_factory=asyncio.Queue)
    seen: list[str] = field(default_factory=list)
    processed: asyncio.Event = field(default_factory=asyncio.Event)

    async def run(self) -> None:
        while True:
            job = await self.queue.get()
            if job is None:
                self.queue.task_done()
                break
            await asyncio.sleep(0.05)
            self.seen.append(job)
            self.processed.set()
            self.queue.task_done()


async def test_worker_records_job() -> None:
    worker = Worker()
    task = asyncio.create_task(worker.run())
    worker.processed.clear()
    await worker.queue.put("invoice-1")
    async with asyncio.timeout(1.0):
        await worker.processed.wait()
    assert worker.seen == ["invoice-1"]
    await worker.queue.put(None)
    await task

python - <<'PY'
import asyncio
from lab_event_wait import test_worker_records_job

asyncio.run(test_worker_records_job())
print("ok")
PY

The timeout context manager needs Python 3.11 or newer, and on older runtimes I would use asyncio.wait_for around the same Event. Would I still use a two-second sleep to let things settle in a test that shares an event loop? Settling is not a condition I can assert, and a condition is something a coroutine can actually await. The timeout is the contract: one second of loop time, then fail closed, then read the task dump. If that feels strict, that is good, because the padded version never was a real test.

What broke along the way

• Module-scoped event loops leaked leftover tasks into the next test and made the Event look already set.
• pytest-asyncio in auto mode hid one file that still created a private loop with asyncio.run.
• A helper named await_ready internally called time.sleep and survived two reviews because the name sounded async.
• asyncio.wait_for on Queue.join passed while seen was still empty, because I joined before the append.

Each of those failures was more educational than the original flake, which only said sometimes. These said you blocked the thread, or you awaited the wrong future, which I can actually fix. Would I go back to rerunning CI to see if the color changes after a nap? Not after this week.

What I would repeat next time

I would start with a single-file reproduction that does not import the application package at all. I would assert on an Event or a future, and I would never assert on the mere passage of time. I would fail the test with a one-second timeout before I ever reached for CI reruns. I would keep assistant output in a scratch file until the local loop proved the helper could fail.

A short checklist I now paste into the PR:

1. Does any test function call time.sleep, os.system("sleep"), or a sync SDK while a loop is running?
2. Is every wait wrapped in asyncio.timeout or an equivalent fail-closed helper?
3. Can I print asyncio.all_tasks() during the wait and still see the worker runnable?
4. If an assistant wrote the helper, did I run the broken reproduction first to prove the helper can fail?

If the answer to item four is no, I do not trust the green check, and neither should you. Is a helper that cannot fail still a test, or is it documentation that happens to be executable? I know which one I shipped for two days, and it was the documentation kind.

Limitations, and who should skip this

This workflow is for in-process asyncio tests where the production bug is that the loop never got a chance to run. It will not explain a hang across processes, a stuck kernel lock, or a real network stall that only appears under packet loss. An Event can still deadlock if nobody sets it and you forget the timeout, which is why the timeout is not optional.

Skip this approach when you are measuring true wall-clock SLAs with a real clock, because asyncio.timeout still uses the loop clock. Skip it when your suite is synchronous and you do not have an event loop to starve in the first place. Skip it when you cannot run the reproduction locally, because a rewrite without a failing proof is just a calmer flake. I am not claiming free model access replaces reading asyncio.all_tasks after you already know the loop is blocked.

The next time a green async test feels too calm, grep for time.sleep before you raise the timeout again. Ask whether the worker could have run during the wait, and then prove it with a timeout that fails closed. Forty-eight hours is a long time to relearn that a thread sleep is not an event-loop scheduler. I would like that lesson to stay in this file instead of in another CI log.