One flaky OTP test can waste a silly amount of team energy. The browser flow looks fine, the API logs say the email was queued, and yet the inbox assertion sometimes grabs an older message and pretends the run is healthy. I kept seeing this in Playwright suites that ran well most days, then got weird when workers, retries, and queue lag lined up just wrong.
The fix that held up best for me was simple: snapshot the inbox before the user action, then compare after the action with a tiny evidence bundle. It is not fancy, but it makes QA decisions much more explainable. When a test fails, you can tell whether no message arrived, the wrong one arrived, or an older message sneaked in and fooled the check.
Why OTP email tests go flaky in parallel runs
OTP and verification tests fail for boring reasons more often than dramatic ones:
- multiple workers reuse the same inbox alias
- a delayed message from the previous run lands late
- the test matches on subject alone
- cleanup jobs remove the clue you actualy needed
- retries pass by finding any recent message
That last one is the sneakiest. A green retry can hide a real app bug if the first attempt already triggered a message and the second attempt simply finds it. Teams then write notes like tem email looked fine or tempail inbox was noisy, which is honest, but not precise enough to debug.
If you test signup or auth flows with a temporary disposable mail provider, this gets even more important. I do not mind using tools like tempmailso in QA, but I never want provider polling to become my only proof.
Take an inbox snapshot before the user action
Before clicking "Send code" or "Verify account", capture a small snapshot of the target inbox state. I only need a few fields:
- recipient address
- current message ids or count
- captured timestamp
- worker id or scenario id
- the event I expect next
This gives me a baseline. After the UI action, I poll again and compare against that baseline instead of searching the full inbox like a loose text query.
That baseline also pairs nicely with request ids in signup email APIs. If the backend already emits a request id, the test can carry it through the browser, API logs, and inbox assertion without much ceremony.
Store a small evidence bundle, not just pass or fail
I like writing one artifact per test attempt. Nothing huge, just enough to review later:
type InboxSnapshot = {
recipient: string;
seenMessageIds: string[];
capturedAt: number;
worker: string;
expectedEvent: "signup-otp" | "login-otp";
};
type InboxCheckResult = {
before: InboxSnapshot;
afterCount: number;
newMessageId: string | null;
receivedAt: string | null;
matchedSubject: string | null;
};
This artifact becomes far more useful than a bare boolean. It also keeps your privacy posture saner, because you can store only the evidence needed to prove the flow worked. That is the same reason I liked reading about short-lived signup log rules: useful debugging data should have clear boundaries.
A Playwright helper that compares before and after
The helper can stay pretty small:
async function waitForNewOtpMessage(snapshot: InboxSnapshot) {
const messages = await inbox.poll({
recipient: snapshot.recipient,
timeoutMs: 20_000
});
const match = messages.find((message) => {
const isNew = !snapshot.seenMessageIds.includes(message.id);
const isFresh = Date.parse(message.receivedAt) >= snapshot.capturedAt;
return isNew && isFresh && /code|otp|verify/i.test(message.subject);
});
return {
before: snapshot,
afterCount: messages.length,
newMessageId: match?.id ?? null,
receivedAt: match?.receivedAt ?? null,
matchedSubject: match?.subject ?? null
};
}
The important part is not the regex. It is the comparison:
- Reject messages already visible in the baseline.
- Reject messages older than the captured timestamp.
- Return evidence the team can read later.
That third step sounds small, but it saves a lot of wasted back-and-forth. A failing artifact lets you see whether the app never sent the OTP, whether the inbox was stale, or whether the subject match was too broad. It is a bit less magical, and that is good.
What to check before blaming the mail provider
When the test fails, I try these checks before raising the timeout or blaming the provider:
- Confirm the app emitted the expected send event for this scenario.
- Verify the inbox alias was unique per worker or per run.
- Compare the new message count against the baseline snapshot.
- Check whether the matched message arrived after the click, not before.
- Make sure the OTP parser is reading the current template, not last month's variant.
- Keep the evidence file beside the Playwright trace so triage stays easy.
Sometimes the provider really is slow, sure. But in my experiance, most flaky OTP tests are self-inflicted: broad matches, shared inboxes, or missing run context. Fix those first and the suite gets calmer fast.
Q&A
Should every test use a fresh inbox?
Not always. A fresh inbox is the cleanest option, but snapshot comparison can still work well when you must reuse an inbox for cost or setup reasons.
Is this useful outside Playwright?
Yes. The pattern is generic QA plumbing. Cypress, API tests, and worker-level smoke tests can all use the same before/after idea.
Do I need to store full email bodies?
Usually no. For OTP checks, metadata and the extracted code are often enough. Smaller artifacts are easier to review and less likely to become accidental long-term storage.
