Skip to content

docs: troubleshoot Browser checks that fail intermittently - #818

Merged
hlenke merged 6 commits into
mainfrom
docs/intermittent-browser-failures
Nov 11, 2026
Merged

hlenke merged 6 commits into
mainfrom
docs/intermittent-browser-failures

Conversation

@hlenke

@hlenke hlenke commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

What changed and why

People often ask why a Browser check fails only sometimes, usually with a timeout. The troubleshooting page had one-line tips, and one of them (use waitForLoadState() for page readiness) is part of the problem. Popups that appear only sometimes (page.addLocatorHandler()), shared beforeAll contexts, redirect races and browser launch errors weren't covered anywhere.

  • Browser checks → Troubleshooting: new Intermittent failures section (after the existing accordions):
    • an intro: what Checkly runs (one worker, no Playwright retries, 30 s default test timeout, a new browser per run), and to start from the failed result's error, call log, trace and video;
    • accordions for hard waits, load/networkidle with hanging third-party scripts, optional popups with page.addLocatorHandler(), shared beforeAll contexts, strict mode violations and hidden duplicates, redirects that race with the next step, browserType.launch: Target page, context or browser has been closed, and failures from one location only.
    • Each accordion uses the existing Issue / Solutions format, quotes the error people see and links to the learn/playwright articles instead of repeating them.
  • Failures that started after a runtime change accordion (in the new section): run the check on the previous runtime with runtimeId and npx checkly test --verbose, check the runtime in the run log, and compare traces. Links to the runtimes page for the web app.
  • Timing and Race Conditions accordion: item 3 no longer recommends waitForLoadState() for page readiness; it points to a web-first assertion and the new section.
  • Error Recovery accordion: the snippet that closed an optional popup with try { click({ timeout: 5000 }) } catch is replaced with page.addLocatorHandler(), as the new section recommends. The try/catch version fails when the popup appears after the catch window.
  • learn/playwright/selectors.mdx, Position-Based Selection: the advice to add .first() "almost every time" is replaced. Use .first() only when you know the first match is the one you want; hidden copies make a CSS or text locator's .first() pick an invisible element. Otherwise make the locator more specific (example: getByRole with a name).

How each claim was verified

The Browser check runner can't run in an isolated local devenv. So each claim was tested with real Playwright 1.58.2, the version in runtime 2026.04, against small local test pages. The runs used the config Checkly's own code generates for a scheduled Browser check on monorepo main (e086a77755):

  • getPlaywrightTestConfig: workers 1, trace and video retain-on-failure, default test timeout;
  • plus the runner settings in browser-runner.ts: retries 0, global timeout 240 s.

Every code sample in the PR was run as written, with only the URLs pointing at local pages.

Claim How it was verified
One worker, no Playwright retries, 30 s default test timeout unless set Generated config (workers: 1, no timeout unless the check sets one) and browser-runner.ts (retries: 0, timeout: inputConfig.timeout). Runs time out at 30000 ms
Each run starts a new browser Code: each run spawns its own playwright test process in a new temp directory (browser-runner.ts)
Scheduled runs keep trace and video only on failure Existing docs (Viewing trace files) and the generated config retain-on-failure
Hard wait too short reads the page too early Run: Expected: "Ready" / Received: "Loading". A wait longer than the test timeout fails with page.waitForTimeout: Test timeout of 30000ms exceeded.. The toHaveText fix passes
A hanging third-party script blocks load A page loading an async script from another host that never responds. page.goto fails with page.goto: Test timeout of 30000ms exceeded. and the call log line navigating to "…", waiting until "load". Run: waitForLoadState('load') reports "domcontentloaded" event fired before it times out
networkidle never settles on a polling page Run: page.waitForLoadState: Test timeout of 30000ms exceeded.. 500 ms definition and "discouraged" from the Playwright docs
Trace lists the request that never finished The failed run's trace.network has the script request with no response (status -1)
waitUntil: 'domcontentloaded' + assertion, or page.route(...abort), fix it Both snippets pass against the hanging page
Popup makes the click time out with intercepts pointer events Run: locator.click: Test timeout … with <div id="overlay">…</div> intercepts pointer events
addLocatorHandler fixes it, and does nothing when no popup Snippet passes with the popup, without it, and with a popup that appears 4 s later
try/catch with a short click timeout fails when the popup comes later Run: the popup appears after the 2 s catch window and the later click times out
addLocatorHandler needs Playwright 1.42 (runtime 2024.02+) Playwright release notes and runtime specification page (2024.02 = 1.42.1); code: runtime 2024-02 package.json
A beforeAll context leaks state between tests and has no video Run: the second test sees 1 items instead of 0 items; the failed result has a trace but no video. With the page fixture both tests pass and a failing test has a video
Strict mode violation text Run: Error: locator.click: Error: strict mode violation: locator('.submit') resolved to 2 elements:
.first() can pick a hidden copy Runs with a hidden mobile-menu button and hidden calendar entries: click times out with element is not visible
getByRole, scoping, and filter({ visible: true }) fix it All three pass. getByRole ignores the hidden duplicate. filter({ visible }) needs Playwright 1.51, so runtime 2025.04 per the release notes and the runtime specification page
A late redirect cancels a click or navigation A page that redirects 1 s after load. A page.goto started just before fails with page.goto: net::ERR_ABORTED at …. A click just before leaves the next assertion failing. Whether it fails depends on timing: in a delay sweep, 950 ms passed and 990 ms failed
toHaveURL / waitForURL before the next step fixes it Both snippets pass
browserType.launch: Target page, context or browser has been closed happens before test code runs, no video Run with Checkly's config and a browser binary that exits on start: exactly this error, test duration 0 ms, the test body's log line never printed, and no video
Retries run the whole check again with a new browser Code: RetryService reschedules the whole check, and each run is a new browser process. Retries docs
Error Recovery: the addLocatorHandler snippet closes the popup when it's there and does nothing when it isn't The accordion's whole snippet run as written against a page whose .optional-popup covers the Log in button: popup shown before the action (handler closes it, click lands), no popup (click lands), popup appearing 6 s after load, past the old 5 s catch window (handler closes it, click lands)
Error Recovery: the old try/catch snippet fails with a late popup The removed snippet run as written against the same 6 s popup: the catch window ends, the later click times out with <div class="optional-popup">…</div> intercepts pointer events
Selectors: .first() on a CSS or text locator can pick a hidden copy and time out locator('.submit').first().click() on a page with a hidden mobile-menu button first, and getByText(…).first().click() on hidden calendar entries: both resolve to the hidden element and time out with element is not visible
Selectors: the existing getByRole('button').first() example and the new getByRole('button', { name: 'Submit' }) example work on that page Both run as written: getByRole skips the hidden copy and clicks the visible button. The article's filtering approach (filter({ visible: true })) also picks the visible entry
A check runs on the runtime set with runtimeId; the run log names it Real run on a production test account (CLI 10.1.0): a Browser check with runtimeId: '2025.04' and npx checkly test --verbose. Log: Creating runtime version 2025.04 using Node.js 22; the script logged Playwright 1.51 and Chromium 134 (2025.04's versions). The same check on 2026.04 logs Creating runtime version 2026.04 using Node.js 24
Rendering mint dev (both changed pages) and the docs checks (frontmatter, anchors, links, redirects, broken links) all pass

Not tested / left out

🤖 Generated with Claude Code

Add an "Intermittent failures" section to the Browser checks troubleshooting
page: hard waits, load/networkidle with hanging third-party scripts,
addLocatorHandler for optional popups, shared beforeAll contexts, strict mode
and hidden duplicates, redirect races, browser launch errors, and
location-specific failures. Stop recommending waitForLoadState() for page
readiness in the timing accordion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
checkly-422f444a 🟢 Ready View Preview Nov 11, 2026, 3:23 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

- Browser check troubleshooting, Error Recovery: replace the try/catch
  pop-up snippet with page.addLocatorHandler(), as the new intermittent
  failures section recommends.
- learn/playwright/selectors: use .first() only when the first match is
  the one you want; hidden copies make it pick an invisible element.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tested runtimeId 2025.04 with checkly test; the run log names the runtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@hlenke hlenke left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready to merge · Every runner claim and Playwright API in the new section checks out against the runner source and Playwright's docs. The four wording items below are optional.

Overlap. #813 also edits detect/synthetic-monitoring/browser-checks/troubleshooting.mdx. The hunks don't overlap: #813 adds a "Missing traces or videos" accordion after Performance and Timeout Issues, and this PR edits the Timing and Race Conditions list and appends a section at the end. Either order merges cleanly; I'd land #813 first. Once both land, the page says twice that pages from browser.newContext() have no video: #813's item 3 and line 179 here. Consider shortening line 179 to a link to that accordion, or keep it, since it explains the beforeAll case.

CI: all checks pass. Mergeable: clean. main has not changed the touched files since the merge base.

Inline: 4 wording.

What I checked
  • Read the full diff from the merge base e0e0277f, every line of both files, and the whole troubleshooting page and learn/playwright/selectors.mdx at the PR head.
  • Local app unavailable (BUSY: another Tilt on the ports), so I verified against source instead of end-to-end runs. The PR description reports runs of every snippet with Playwright 1.58.2 and a production --verbose run for the runtime log line. I didn't repeat those.
  • Runner settings in the intro (one worker, no Playwright retries, default test timeout, per-run browser, retain-on-failure for scheduled runs): monorepo main packages/playwright-check-runner/src/browser/browser-runner.ts and packages/check-runtimes-common/src/browser/getPlaywrightTestConfig.js. The test timeout is taken from the check's playwrightConfig.timeout when set, which is why I suggest mentioning it on line 94. playwright-support.mdx lists timeout as a supported global option.
  • Playwright APIs: page.addLocatorHandler() added in v1.42 and runs before actionability-checked actions and auto-waiting assertions (https://playwright.dev/docs/api/class-page#page-add-locator-handler). networkidle is DISCOURAGED and means no network connections for at least 500 ms (same page, waitUntil). locator.filter({ visible }) added in v1.51 (https://playwright.dev/docs/release-notes). getByRole excludes hidden elements by default.
  • Runtime mapping: platform/runtimes/runtime-specification.mdx lists Playwright 1.42.1 for 2024.02 and 1.51.1 for 2025.04.
  • Code samples: read every block for valid Playwright. No await on locators, locator-based actions, @playwright/test style. Single quotes and no semicolons match the rest of the page.
  • Links and anchors: #viewing-trace-files, #avoid-network-interception-in-synthetic-monitoring, #handling-multiple-matching-elements, and #overriding-runtimes-for-specific-checks exist on main. scripts/pr_impact.py: no findings.
  • Not verified: the browserType.launch cause on Checkly's runners, beyond the author's simulated run.

@hlenke
hlenke enabled auto-merge (squash) November 11, 2026 03:23
@hlenke
hlenke merged commit 5fee8df into main Nov 11, 2026
7 checks passed
@hlenke
hlenke deleted the docs/intermittent-browser-failures branch November 11, 2026 03:25

This branch was successfully deployed

1 active deployment
staging — e2e1185a Deployed Nov 11, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant