Avoiding flaky OTP timing in Selenium email tests

The design rule

Flaky OTP timing in Selenium almost always comes from fixed Thread.sleep assumptions about email arrival. Replace sleeps with an explicit wait loop that polls your inbound source until a message matching the run ID appears—or until a bounded timeout fails the test with a clear error.

Do this first: give every test run a unique address or subject token; wait on that signal; assert the OTP; tear down the inbox and browser session. Use Mailby Quick Inbox for manual QA spikes. For automation, prefer the live developer API or a mail catcher you own—do not scrape a consumer session cookie in CI as if it were a stable API.

Selenium context and boundaries

Selenium drives browsers. Email lives outside the DOM. Coupling them without a contract produces races:

• ESP latency spikes
• App-side queue delay
• DNS cold starts in ephemeral environments
• Parallel tests stealing each other’s codes

Boundaries:

• Test only apps you own or have written permission to test.
• Prefer staging.
• Mailby receive-only inboxes are tools, not senders.
• Retention clocks will kill messages mid-debug if you walk away—see data retention.

Selenium’s own guidance emphasizes explicit waits over implicit sleeps—see Selenium waits documentation (rel="nofollow noopener").

Pattern that removes most flakes

Test date: 2026-09-24. Pattern: unique inbound mailbox per test + poll + assert + cleanup.

1. Provision an inbox (API or catcher) tagged with testRunId.
2. Drive UI to request OTP for that address.
3. Poll GET messages until subject/body contains testRunId or OTP regex, every 2–5s, max 60–120s (tune to your SLA).
4. Extract OTP; enter into browser; wait for post-login element with WebDriverWait.
5. Teardown inbox, user, and WebDriver—always, including on failure.

Working path: p95 email arrival 8s; timeout 90s; flake rate near zero for timing reasons.

Failure / limitation (counterexample): a suite used sleep(5) because “email is always fast on localhost Mailpit,” then moved to a cloud ESP. Arrival p95 became 20s. Tests failed red on timing while the app was healthy. Sleeps hid the real SLA; bounded polls surface it.

Mechanism

Flakiness formula:

success = (email_arrival_time + browser_input_time) < hard_sleep_budget

When arrival is a distribution, fixed sleeps either waste time or fail the right tail. Event waits track the distribution’s signal instead of its hope.

Also flake sources mistaken for timing:

• Reusing one shared inbox across parallel workers
• Matching the wrong message (oldest vs newest)
• Clock skew on OTP expiry while the wait loop crawls

Table and worked example

Test step	Timeout / retry	Assertion	Teardown
Create inbox	Fail fast if API down	Inbox ID returned	Delete inbox
Submit signup	WebDriverWait for success toast	Request accepted	—
Wait for email	Poll ≤90s, 3s interval	Message with run ID	—
Extract OTP	Regex must match once	6-digit code	—
Enter OTP	Wait for dashboard root	URL / element visible	—
Expire unused mail	—	—	Purge inbox / revoke tokens

Worked example — Priya’s CI

Priya’s Selenium suite created qa+<uuid>@catcher addresses. The wait helper threw EmailNotArrived(runId, waited=90) with the last SMTP log snippet attached. Flakes labeled “OTP” dropped after shared-inbox races were removed—half the “timing” failures were collisions.

Manual exploratory passes still used Quick Inbox outside CI.

Alternatives

• Bypass OTP in test builds behind a feature flag (fastest unit of UI work).
• Mailpit/Mailhog for local.
• Mailby developer API for hosted inbound waits—/developers.
• Durable QA mailboxes for human nightly checks—not temporary consumer sessions for parallel CI.

Related: how it works, pricing if retention during long manual debug matters.

Short answers

What causes flaky OTP timing in Selenium?
Fixed sleeps, shared inboxes, and unclear timeouts—not Selenium itself.

What should I do first?
Unique mailbox per run + explicit poll with a measurable timeout.

When is a permanent address safer?
Long-lived human QA accounts; never share them across parallel jobs.

What evidence changes the recommendation?
ESP p95 exceeds your product’s OTP TTL—you must fix server TTL or queueing, not the wait helper.

Sources, limitations

• Selenium waits docs (nofollow).
• Editorial pattern 2026-09-24.
• Exact API shapes differ; adapt to your catcher.

Reference wait helper (shape, not vendor lock-in)

Pseudocode you can adapt:

deadline = now + 90s
while now < deadline:
  messages = inbound.list(inboxId)
  match = first message where runId in subject or body
  if match:
    return extract_otp(match.body)
  sleep 3s
throw EmailNotArrived(runId, waited=90)

Pair with WebDriverWait for the post-login element. Never sleep(90) “just in case.”

Log the wait duration on success. If p95 climbs week over week, file an app/ESP ticket—the test is now a probe.

Parallelism rules

• One inbox per worker minimum; prefer one inbox per test method.
• Do not search “latest OTP” globally.
• Namespace subject lines: [otp][runId].
• Disable retries at the Selenium runner level until inbound waits are stable—double retries amplify mail storms.

Manual QA vs CI

Manual explorers can use Quick Inbox with human patience. CI must not drive a headed browser into a consumer session cookie. Use API-accessible inbound (developers) or a catcher in the cluster network.

Measuring “fixed”

Track:

• OTP-related flake rate / week
• p50/p95 inbound wait
• Count of EmailNotArrived errors

A change that only adds longer sleeps will look greener for a week and then fail harder under load. Bounded waits keep the signal honest.

How this differs from an email-test-automation hub

Hubs list tools. This page is an implementation pattern for flaky OTP timing under Selenium, with a collision counterexample and a step/timeout/assert/teardown table.

Separating browser flakiness from mail flakiness

Not every red OTP test is email. Element locators break, animations eat clicks, and headless timing differs from headed. Tag failures:

• EMAIL_TIMEOUT — wait helper exhausted
• OTP_REJECTED — code entered but UI error
• DOM_STALE — Selenium locator issues

Only the first category should trigger ESP investigations. Mixing them wastes vendor goodwill.

Seeded OTPs in test builds

Where security review allows, test builds may accept a magic OTP for specific users. That removes email from UI tests entirely. Keep a smaller suite that still exercises real mail nightly so template and delivery regressions are caught. Balance speed and coverage deliberately.

Local vs CI network egress

CI runners may be IP-blocked by aggressive ESPs. If inbound never arrives in CI but works on laptops, check deny lists. A receive-only API in the same VPC as your app can be more reliable than public disposable domains from cloud IPs.

Human fallback protocol

When CI email is down, do not silently skip security tests. Fail the pipeline or run a labeled manual path with Quick Inbox and attach screenshots. Skipping OTP tests for a week is how broken reset templates ship.

Case study narrative: fixing a 22% flake rate

A SaaS QA team saw 22% of signup Selenium tests fail on OTP entry. Initial “fix” increased sleep from 5s to 25s. Failures dropped to 9% and pipeline time ballooned. EmailNotArrived was never distinguished from wrong-code errors.

Investigation showed three causes: (1) shared catch-all inbox across four parallel workers, (2) p95 ESP latency of 18s against a 15s sleep, (3) tests grabbing the oldest unread message.

Remediation: unique inbox per test via API; poll 90s; select newest message matching runId; tag failures. Flake rate fell below 1% for timing classes. Remaining failures were genuine app bugs—exactly what CI should surface.

They kept a weekly manual path with Quick Inbox to eyeball template regressions that API assertions might miss (broken CSS, misleading copy). Manual was exploratory, not the parallel CI backbone.

Secondary improvement: product extended OTP TTL from 5 to 15 minutes after seeing wait metrics. Tests had become production telemetry. That is the healthy end state—automation as sensor, not as a sleep cushion.

If your org cannot yet host a catcher, the Mailby developer surface at /developers can provide inbound wait APIs for authorized tests of apps you control. Still apply unique mailbox discipline; a shared consumer-style session will recreate the 22% story.

Publish the wait helper as an internal library so every squad does not reinvent sleeps. Code review guideline: reject new Thread.sleep around mail unless accompanied by a linked issue explaining why polling is impossible.

Anti-patterns library (reject in code review)

• Thread.sleep(5000) after clicking “Send code”
• Shared QA_INBOX env var across parallel jobs
• Parsing “the latest email” without runId
• Retrying the whole Selenium class three times to “soak” flakes
• Disabling OTP in production via the same flag used in tests
• Scraping Mailby consumer cookies from CI browsers

Replace each with unique inbox + poll + explicit assertion + teardown. Document timeouts next to product OTP TTL so they cannot drift silently. Use /developers when you need API waits; use /inbox for manual exploration only.

Coordinating OTP tests with product analytics

Product analytics often already record “OTP sent” and “OTP success” events with timestamps. Join those events to CI wait durations. If analytics say sent→success is 4s p95 for real users but CI waits 40s, your environment—not users—is slow. Fix CI network or ESP sandbox. If both are slow, users feel it too; raise the priority. Selenium should not be the only telemetry, but it is a valid probe when tagged correctly. Share weekly p95 wait in engineering forums so mail is treated as a dependency with SLOs, not as flaky witchcraft. Pair with how it works when onboarding new QA hires to Mailby’s receive-only model so they do not request send/forward features the product will not provide.

Environment matrix worth maintaining

Track results across: local Mailpit, staging ESP, CI runner region A, CI runner region B. Timing distributions differ. A helper tuned only on laptops will fail in CI. Publish the matrix in the QA README with last-updated dates. When Mailby developer inboxes are in the mix, note their region assumptions too. The goal is predictable waits, not identical waits. If region B is always slower, set timeout from the worse environment and keep assertions identical. Humans using /inbox should not be part of the matrix for parallel jobs; they remain exploratory only. This matrix section is the operational companion to the wait helper earlier in the article—without it, teams relearn the same flake every quarter.

Closing engineering guidance

OTP email is an asynchronous dependency. Design for it with unique signals, bounded polls, clear failure taxonomy, and teardown that always runs. Reject sleeps in review. Measure waits. Keep manual Quick Inbox exploration separate from CI. When product TTL is shorter than delivery p95, fix the product or the ESP—not the test harness alone. Align your team on that philosophy and the flake class disappears from standup folklore. Then Selenium can go back to finding real UI bugs instead of racing the mail truck.

Conclusion

Treat inbound OTP as an asynchronous dependency with a contract: unique signal, poll, assert, teardown. Keep Selenium waits explicit. Use Quick Inbox for manual QA and /developers when you automate—without sleeping your way to intermittent red builds.