Avoid flaky OTP timing in Playwright email tests

Replace waitForTimeout(5000) with a contract. Flaky OTP timing in Playwright almost always comes from racing the email pipeline: the test types a code before mail arrives, resends while a previous code is still valid, or tears down the mailbox before assertion. Fix the wait policy, assert on a measurable condition, and clean up. Mailby Quick Inbox is fine for manual QA; for automation, use deterministic app hooks or the live developer API—do not scrape a disposable web UI as your only CI strategy.

Playwright context and boundaries

You are testing an application you own. Email may be:

• Intercepted in-process (test double)
• Delivered to a catcher (Mailpit, etc.)
• Delivered to a real receive-only address

Boundaries:

• Fixed sleeps hide product slowness and create CI landmines.
• Resend buttons invalidate OTPs; tests must be single-flight.
• Mailby does not claim a magic “Playwright plugin”; integrate via documented APIs and manual receive as appropriate.
• Temporary consumer inboxes expire—bad for long parallel suites unless leases match runtime (data retention).

Playwright’s auto-waiting model is documented at Playwright locators.

Pattern: wait on evidence, not on hope

Working path

1. Trigger signup/login once.
2. Poll an inbox API or test seam until a message matching subject/from appears or timeout budget exceeds (for example 30–60s with backoff).
3. Parse OTP with a strict regex anchored to your template (/\b(\d{6})\b/).
4. Fill the OTP field; click continue once.
5. Assert landing URL or authenticated cookie.
6. Teardown: invalidate tokens, delete test user, release inbox.

Illustrative sketch (structure only):

// Pseudocode — wire to your mail catcher or Mailby developer API
const code = await expect.poll(async () => {
  const msg = await mailClient.latest({ to: testEmail, subject: /verify/i });
  return msg?.extractOtp?.() ?? null;
}, { timeout: 45_000 }).not.toBeNull();

await page.getByLabel(/code/i).fill(code!);
await page.getByRole("button", { name: /continue/i }).click();
await expect(page).toHaveURL(/dashboard/);

Failure / limitation observed

Teams that open Quick Inbox in a headed browser inside CI see session/cookie fragility and lease expiry under load. That is a misuse of a human inbox UX. Prefer API access from /account/developer or an in-cluster catcher.

Mechanism: why OTP tests flake

Cause	Mechanism	Symptom
Sleep too short	Mail slower than sleep	Empty input / invalid code
Sleep too long	Masks regressions	Green but slow suite
Double resend	Server rotates secret	Stale code from first mail
Shared inbox	Parallel workers steal mail	Intermittent cross-talk
Clock skew	Token exp vs runner time	Valid-looking code rejected
UI preview delay	Human inbox rendering	Automation timeout

Step table

Test step	timeout/retry	assertion	teardown
Submit signup	Playwright default nav timeout	200/302 or confirmation UI	—
Await message	Poll ≤45s exponential backoff	Message present for to	—
Extract OTP	0 retries on parse fail	Regex match length	—
Submit OTP	1 attempt	Authenticated state	Invalidate unused tokens
Negative: wrong OTP	Immediate	Error banner	Clear field
Worker isolation	N/A	Unique address per test	Delete inbox/user

Worked example

A Playwright suite slept 3 seconds, then read a Mailhog message. Under load, P95 delivery was 4.2s → 15% flake. Switching to expect.poll on Mailhog’s API with a 45s budget dropped flakes below 1% and surfaced a real SMTP misconfig when mail never arrived by budget end.

Alternatives

• Bypass seam: test endpoint that returns the OTP in non-production.
• Local SMTP catcher: Mailpit/Mailhog in docker-compose.
• Mailby developer API: real MX path without owning SMTP (/developers).
• Manual Quick Inbox: exploratory tests only.

Durable shared QA mailboxes help long-lived accounts; unique per-test addresses prevent collisions.

Short answers

What causes flaky OTP timing in Playwright?

Racing delivery, invalidating resends, and shared inboxes—not Playwright itself.

What should I do first?

Remove fixed sleeps; poll for the message with a clear timeout; forbid parallel access to one address.

When is a permanent address safer?

For human QA personas. Automation should prefer ephemeral unique addresses with known retention.

What evidence changes the recommendation?

If your app cannot expose a test seam and public temp-mail domains are blocked, you need a first-party catcher or developer inbox API.

Sources, test date, limitations

• Patterns reviewed 2026-09-24.
• Playwright documentation.
• OTP delivery still rides SMTP (RFC 5321).

Limitations: Code is illustrative. Distinct from a generic automation hub by focusing on OTP timing contracts.

Conclusion

Flake is a timing contract bug. Wait for message evidence, submit once, tear down hard. Use Quick Inbox for manual checks; use /developers when you need programmable receive paths. Align lease length with suite duration via pricing and data retention.

Budget math for OTP waits

If P95 email latency in staging is 12s, a 5s sleep fails often and a 60s fixed sleep wastes CI. Prefer poll intervals of 1–2s with a 45–60s budget, and record the observed wait as a metric. When the budget expires with no mail, fail with “mail not delivered” rather than “OTP invalid,” so on-call knows which subsystem broke.

Isolation patterns

• Unique address per worker: qa+{worker}-{uuid}@… or unique Mailby inbox per test
• Never share one Quick Inbox session across parallel shards
• Mutex resend: only the test that owns the account may click resend
• Seed data cleanup in finally / fixtures, including partial failures

Seeded OTP seams (when allowed)

In non-production, a sealed test endpoint that returns the current OTP for a given user beats reading mail entirely. Guard it with environment checks and auth suitable for CI secrets. Use real mail paths in one nightly job so SMTP regressions still surface.

Playwright-specific tips

• Use getByLabel / roles over brittle CSS
• Avoid page.waitForTimeout
• Prefer expect.poll or locator auto-waiting against your mail client helper
• Trace viewer on failure: include the mail API response body (redact secrets in shared logs)

Manual vs automated

Exploratory QA can use /inbox. Automated pipelines should use programmable receive (/developers) or local catchers. Mixing the two without intent causes flake narratives that blame Playwright for product design issues.

Failure taxonomy for triage

When a job fails, classify before retrying:

• Mail never arrived → SMTP/ESP/inbox API issue
• Mail arrived late beyond budget → raise budget or fix sender latency
• Stale OTP → resend race / shared inbox
• UI not ready → Playwright locator issue, not email
• Env mismatch → wrong mail catcher host in CI secrets

Misclassification leads to cargo-cult sleeps.

Contract tests between app and mail helper

Define a small interface:

waitForOtp({ to, subject, timeoutMs }) -> string

Implementations: Mailpit, Mailby developer API, stub. Playwright specs depend on the interface only. Swapping implementations should not rewrite every test.

Parallelism ceilings

If your ESP rate-limits outbound mail, unbounded Playwright shards create false flakes. Cap workers or pre-seed slower environments. Unique inboxes per test remain mandatory even when rate limits are fine.

Human fallback

When automation cannot see mail, a documented manual path using /inbox unblocks release trains—but label it manual so nobody “automates” it with screenshots of a consumer UI. Prefer API access for anything recurring (/account/developer).

Sample timing report to paste into CI summaries

Emit JSON lines:

{"event":"otp_wait_ms","ms":8321,"to":"…","ok":true}

Track P50/P95 weekly. When P95 climbs, fix mail infrastructure before raising timeouts again. Timeouts are a budget, not a strategy.

Local developer experience

Docker Compose with Mailpit + Playwright should be the default happy path for engineers. Remote Mailby API tests prove MX realism but should not be required for every save-file loop. Document both in the README with clear labels: local catcher vs hosted receive (/developers).

Quarantine mode

On flake spikes, quarantine the email suite separately from UI-only suites so product UI regressions still gate merges. Restore the email suite when mail P95 recovers—do not delete coverage permanently.

Full reference sequence (copy into ADR)

1. Provision unique inbox (API or catcher).
2. Create user with that email via API seed (skip UI if possible).
3. Navigate to login; request OTP once.
4. waitForOtp with 45s budget, 1s poll.
5. Fill OTP; assert dashboard.
6. Negative test in separate spec: wrong OTP shows error without requesting new mail.
7. Teardown inbox + user even on failure.

Any deviation (shared inbox, double resend, fixed sleep) must be justified in code review comments. Mailby Quick Inbox remains the manual escape hatch (/inbox), not the CI backbone.

Why fixed sleeps fail math

Suppose delivery latency is normally distributed with mean 4s and stdev 2s. A 5s sleep fails a large tail; an 8s sleep still fails some; a 30s sleep mostly passes but hides a 25s regression that users feel. Polling until evidence appears converts the wait into a measurement. Emit that measurement. When mean drifts from 4s to 15s, alert on the metric—your Playwright red build is a symptom of mail latency, not a reason to delete the test.

Guardrails in code review

Reject PRs that introduce waitForTimeout near OTP flows. Reject shared mailbox fixtures. Require unique addressing. Require teardown. These social rules prevent flake more effectively than another blog post. Keep Mailby links in the README for manual QA (/inbox) and API automation (/developers).

Closing guidance

OTP flake is almost never mysterious. It is a missing wait condition, a shared inbox, or a resend race. Encode the wait as polling against mail evidence, isolate addresses per worker, forbid fixed sleeps, and tear down hard. Use local catchers for day-to-day Playwright loops and hosted receive APIs when you must exercise real MX paths (/developers). Keep /inbox for exploratory QA so CI stays deterministic.

Note on scope

This guide stays within authorized testing and published Mailby product behavior—receive-only inboxes, documented retention, and live developer tools—without promising universal deliverability or anonymity. Verify live UI details on the product pages before you rely on a specific lease length in production workflows for teams you support beyond this article's examples and checklists here.

Appendix: anti-patterns checklist

• Sleeping a fixed five seconds before reading mail
• Clicking resend in a loop until something appears
• Reusing one QA address across ten Playwright shards
• Parsing OTP with a greedy regex that captures order numbers
• Leaving inboxes alive so tomorrow’s run reads yesterday’s code
• Scraping the consumer Quick Inbox DOM as if it were a stable API

Replace each anti-pattern with the positive control described earlier. When mail is genuinely slow, surface latency metrics to the team that owns SMTP—not only to QA. Hosted receive for automation belongs on /developers; human exploration belongs on /inbox.