Repository navigation
feat(clock): core's clock writes wait for the computers on the clock and fail their step naming any behind (FD-5473) - #672
Closed
earakely-scale wants to merge 2 commits into
Conversation
…and fail their step naming any behind (FD-5473) Every clock write core makes goes through DeployedEnv.write_clock: sync_env_clock's arm, the always-on step's freezes, thaws and advances, and a post-turn hook's. It passes wait_followers_s (CLOCK_WAIT_FOLLOWERS_S, 2 s), so the gateway answers once each computer following its clock shows the change, and a 504 raises ClockFollowersBehind naming the computers still behind; the write itself has applied, so it isn't retried, and the step fails. A clock no computer follows answers at once, as before. - The gateway's 504 carries the protocol's error shape (code clock_followers_behind, applied yes) beside its stale and followers, so the client reads its code and message instead of a body of its own. - The always-on step neither retries such a write nor swallows it where it tolerates other failures of an advance. - A hook that catches the error still fails its step: HookEnv keeps it, and the runner raises it after the hook. - An integration test pauses a multi env's computer: core's advance fails after its wait naming it, and once the computer is back the next is acknowledged within the wait, the session on the new time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…owers' 504 read as computers behind Such a gateway answers a write its computers didn't show with a 504 whose error is its message, so the client saw no code and write_clock raised the plain error: the always-on step would retry or swallow an applied write, and a hook's catch would hide it (Greptile). Any 504 whose JSON names its stale computers now counts, whatever its error; a proxy's own 504 page stays the error it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Deferring this for now: it adds complexity we don't need yet, and nothing on main depends on it. Without it, core's clock writes don't wait for the computers on the clock, and a computer that falls behind isn't reported. If that turns out to be a problem (for example on Modal, FD-5487), reopen this PR. The branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
The agent must never act on a computer that still shows the old time. Since #614 a clock write can wait for the computers that follow the gateway's clock (
wait_followers_s), but core never asked it to. Now every clock write core makes does, and a computer that doesn't show the change in time fails the step that moved the clock, naming it (FD-5473, HB1.3).DeployedEnv.write_clock(method, params), forset_time,set_rate,advanceandclear(CLOCK_WRITES). It passeswait_followers_s=CLOCK_WAIT_FOLLOWERS_S(2 s), unless the caller names its own. A 504 raisesClockFollowersBehind, carrying the env, the method and the computers behind (stale). The write itself has applied. A clock no computer follows answers at once, as before.sync_env_clock's arm;HookEnv.invoke, such as theadvance-to-next-eventexample's._callretries other failures, but not a clock write the computers missed, since it applied (a repeatedadvanceto the same time would only be refused). Itsadvanceand settle no longer swallow it where they tolerate other failures.advance. SoHookEnvkeeps aClockFollowersBehindit raised, and the runner raises it once the hook returns, caught or not.error: {code: "clock_followers_behind", message, applied: "yes", retryable: false}), beside itsstaleandfollowers, so the client reads its code and message (ExtensionCallError) instead of a body of its own. Before, itserrorwas a string, which the client couldn't read, and a step would have said only "504 Gateway Timeout". A gateway from before this change still answers that way, sowrite_clockcounts any 504 whose JSON namesstalecomputers as behind, whatever itserror; a proxy's own 504 page stays the error it is.Behaviour to know about
erroris an object now,staleandfollowersare unchanged). Nothing in the repo read the old string but its own test.Testing
tst/unit/env/write_clock_test.py(new), against a gateway in this processsync_env_clock_test.pytest_always_on_clock_followers.py(new), the always-on rig with a real gateway that declares a computertest_prompt_agent_post_turn_hooks.pycomputer_clock_test.py(new test), a multi env's computer on local Docker🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; the older-gateway compatibility gap is fixed.Summary
Core clock writes now wait for computers to show the new time before the agent continues.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Clock write from a step or hook] --> B[DeployedEnv.write_clock] B --> C[Gateway applies the write] C --> D{Computers show the change?} D -->|Yes| E[Step continues] D -->|No| F[504 names computers behind] F --> G[ClockFollowersBehind] G --> H[Step fails without retrying]Reviews (6) · Last reviewed commit: "fix(clock): a gateway from before the er..." · Reviewed by Greptile