Skip to content

fix(computers): checks that failed on a busy runner give it room, or drop what only timing decided - #670

Merged
earakely-scale merged 4 commits into
mainfrom
edgararakelyan/busy-runner-flakes
Nov 13, 2026
Merged

earakely-scale merged 4 commits into
mainfrom
edgararakelyan/busy-runner-flakes

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

Checks that failed on main and on unrelated branches over the last day and a half of local-backends runs, and on CodeBuild, each on a busy runner. Each either gets room it needed, or loses the part only timing decided:

Check Failed with Change
The gate-3 probe's recording check (conformance/checks.py, _segments_match) "8 frames over 1.40 s of a 2 s recording at 5 fps" (main e5c1ab9), "9 frames over 1.59 s" The segments must span half the recording, as the frames must already number half. A recorder that takes a moment to start no longer fails gate 3; one that stalls after its first frame still does (test_a_stream_that_stalls_after_its_first_frame_fails). A probe check that fails fails the compose, and so every test that needed that computer: computer_app_test's WebSocket test failed this way on main.
page_driver_test test_a_call_is_bounded_from_when_it_asks_for_its_session_and_leaves_no_session_half_open a gateway session owner (ChildSessions._own) still running 0.1 s after the bounded call (main e5c1ab9 and 3 other branches) Waits up to 5 s for the owners to end (OWNERS_END_WITHIN_S, polling every OWNERS_POLL_S), so one that never ends still fails.
test_computer_app_server test_the_drivers_readiness_asks_every_apps_server_at_once 3.15 s with 2.5 s allowed, on CodeBuild (#666) Times the readiness request alone, not the fake driver's setup, and each app's check takes 2 s: asked at once the status takes about 2 s, one after another 6 s, and the bound is 4 s. The driver does ask at once (runtime.ready_checks gathers them).
desktop_example_test test_a_scripted_drive_is_recorded_on_the_clock_its_screenshots_carry the clicked cell's first changed frame 0.137 s after the screenshot that showed it, 0.1 s allowed (main 8668b2b); and on #666, no recorded frame showing the change at all (StopIteration) The click-repaint timing is gone: it failed on 5 commits, both ways down to when the recorder writes a frame under load. The test still checks each screenshot against the recording's nearest frame, in time and in pixels, which is what puts the two on one clock.
desktop_example_test test_the_desktop_examples_computer_passes_the_probe_at_full "scored below 1 (probe: 0)" on #666, with no word of which check Its failure now names each probe check that failed and why, from the run's verification.

Testing

  • page_driver_test.py, test_computer_conformance_recording.py and the readiness test pass locally.
  • desktop_example_test.py runs in integration-local-slow on this PR.

Not in this PR

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

This PR appears safe to merge.

Summary

This PR gives busy runners more room without removing the remaining recording and cleanup checks.

  • The recording probe allows startup delay without skipping frame checks.
  • Failed desktop probes show which checks did not pass.
  • Desktop recording checks no longer time click repaint against video frames.
  • The readiness test distinguishes parallel checks from slow setup.
  • The page-driver test gives session owners more time to finish.

The previous unnumbered polling-delay finding is fixed.

Reviews (6) · Last reviewed commit: "test: the desktop example's scripted dri..." · Reviewed by Greptile

From the last day and a half of local-backends runs, main's included:

- The probe's recording check required a 2 s recording's segments to span 1.6 s; a recorder that took a moment to
  start spanned 1.40 s and 1.59 s, failing gate 3, so whichever test composed that computer failed
  (computer_app_test, website_browser_test). It now requires half the recording, as it does of the frames, which a
  stream that stalls still fails.
- desktop_example_test required the recording's first frame showing a clicked cell within one frame interval of the
  screenshot that first showed it; a frame written 37 ms late failed it. It allows two intervals.
- page_driver_test checked that no session owner was left 0.1 s after a bounded call; it now waits up to 5 s for them
  to end, so one that never ends still fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 13, 2026 18:42
Comment thread tst/unit/drivers/page_driver_test.py Resolved
earakely-scale and others added 2 commits November 13, 2026 11:10
…against one-after-another asks

On CodeBuild (#666's Integration Tests), test_the_drivers_readiness_asks_every_apps_server_at_once took 3.15 s with
2.5 s allowed: its clock started before it built the fake driver's client, and three 1 s app checks asked at once left
only 1.5 s for that on a busy machine. The driver does ask them at once (runtime.ready_checks gathers them). The clock
now starts at the request, and each app's check takes 2 s, so asked at once the status takes about 2 s, one after
another 6 s, and the bound is 4 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale earakely-scale changed the title fix(computers): three checks that failed on a busy runner give it room fix(computers): checks that failed on a busy runner give it room, or drop what only timing decided Nov 13, 2026
…hout the click-repaint timing, and its probe test names what failed

The scripted drive's click-repaint assertion failed on 5 commits, main included, two ways: the first recorded frame
showing the clicked cell came 37 ms past the frame allowed, and on #666 no recorded frame showed the change at all.
Both come down to when the recorder writes a frame under load. The test still checks each screenshot against the
recording's nearest frame, in time and in pixels, which is what puts the two on one clock.

The probe test failed on #666 with "scored below 1 (probe: 0)" and nothing more; it now names each probe check that
failed and why, from the run's verification.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 0faae73 into main Nov 13, 2026
37 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/busy-runner-flakes branch November 13, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant