Skip to content

feat(clock): core's clock writes wait for the computers on the clock and fail their step naming any behind (FD-5473) - #672

Closed
earakely-scale wants to merge 2 commits into
mainfrom
edgararakelyan/fd-5473-hb13-core-waits-sync_env_clock-post
Closed

earakely-scale wants to merge 2 commits into
mainfrom
edgararakelyan/fd-5473-hb13-core-waits-sync_env_clock-post

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

The agent must never act on a computer that still shows the old time. Since #614 a clock write can wait for the computers that follow the gateway's clock (wait_followers_s), but core never asked it to. Now every clock write core makes does, and a computer that doesn't show the change in time fails the step that moved the clock, naming it (FD-5473, HB1.3).

  • One path for core's clock writes: DeployedEnv.write_clock(method, params), for set_time, set_rate, advance and clear (CLOCK_WRITES). It passes wait_followers_s = CLOCK_WAIT_FOLLOWERS_S (2 s), unless the caller names its own. A 504 raises ClockFollowersBehind, carrying the env, the method and the computers behind (stale). The write itself has applied. A clock no computer follows answers at once, as before.
  • Its callers:
    • sync_env_clock's arm;
    • the always-on step's freezes, thaws and advances, including its advance-to-now while settling;
    • a post-turn hook's clock writes through HookEnv.invoke, such as the advance-to-next-event example's.
  • Not retried, not swallowed: the always-on step's _call retries other failures, but not a clock write the computers missed, since it applied (a repeated advance to the same time would only be refused). Its advance and settle no longer swallow it where they tolerate other failures.
  • A hook can't hide it. Hooks are task code, and both example hooks catch any failed advance. So HookEnv keeps a ClockFollowersBehind it raised, and the runner raises it once the hook returns, caught or not.
  • The gateway's 504 now carries the protocol's error shape (error: {code: "clock_followers_behind", message, applied: "yes", retryable: false}), beside its stale and followers, so the client reads its code and message (ExtensionCallError) instead of a body of its own. Before, its error was a string, which the client couldn't read, and a step would have said only "504 Gateway Timeout". A gateway from before this change still answers that way, so write_clock counts any 504 whose JSON names stale computers as behind, whatever its error; a proxy's own 504 page stays the error it is.

Behaviour to know about

  • A step that moves the clock now takes up to 2 s longer when a computer is slow to show the change, and fails when one doesn't. With no computer on the clock, nothing changes.
  • The gateway's 504 body changed shape (error is an object now, stale and followers are unchanged). Nothing in the repo read the old string but its own test.

Testing

Test What it shows Result
tst/unit/env/write_clock_test.py (new), against a gateway in this process each write asks to wait; a clock no computer follows answers at once; a computer that never shows the change raises naming it, after the wait, with the change applied; a caller's own wait stands; another refusal stays the gateway's error; an older gateway's 504 naming computers behind counts the same; a proxy's 504 page doesn't 6 passed
sync_env_clock_test.py the arm asks to wait; an arm a computer didn't show fails the step naming it, before any server syncs passed
test_always_on_clock_followers.py (new), the always-on rig with a real gateway that declares a computer the step fails naming it, with the clock changed once: no retry passed; fails without the step's changes
test_prompt_agent_post_turn_hooks.py a hook that catches the error still fails the step, and the agent isn't sent its message passed; fails without the runner's check
The touched unit files together 1172 passed
computer_clock_test.py (new test), a multi env's computer on local Docker with the computer paused, core's advance fails after its wait, naming it; unpaused, the next is acknowledged within the wait and the session is on the new time 2 passed on an x86 host: paused, it failed after 2.02 s; back, the next was acknowledged in 17 ms (the prototype's 2.0 s and 16 ms)

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; the older-gateway compatibility gap is fixed.

Summary

Core clock writes now wait for computers to show the new time before the agent continues.

  • Clock writes wait for computers to show the new time.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Clock write from a step or hook] --> B[DeployedEnv.write_clock]
  B --> C[Gateway applies the write]
  C --> D{Computers show the change?}
  D -->|Yes| E[Step continues]
  D -->|No| F[504 names computers behind]
  F --> G[ClockFollowersBehind]
  G --> H[Step fails without retrying]
Loading

Reviews (6) · Last reviewed commit: "fix(clock): a gateway from before the er..." · Reviewed by Greptile

…and fail their step naming any behind (FD-5473)

Every clock write core makes goes through DeployedEnv.write_clock: sync_env_clock's arm, the always-on step's freezes,
thaws and advances, and a post-turn hook's. It passes wait_followers_s (CLOCK_WAIT_FOLLOWERS_S, 2 s), so the gateway
answers once each computer following its clock shows the change, and a 504 raises ClockFollowersBehind naming the
computers still behind; the write itself has applied, so it isn't retried, and the step fails. A clock no computer
follows answers at once, as before.

- The gateway's 504 carries the protocol's error shape (code clock_followers_behind, applied yes) beside its stale
  and followers, so the client reads its code and message instead of a body of its own.
- The always-on step neither retries such a write nor swallows it where it tolerates other failures of an advance.
- A hook that catches the error still fails its step: HookEnv keeps it, and the runner raises it after the hook.
- An integration test pauses a multi env's computer: core's advance fails after its wait naming it, and once the
  computer is back the next is acknowledged within the wait, the session on the new time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 13, 2026 19:43
Comment thread src/agent_env/env/env.py Resolved
…owers' 504 read as computers behind

Such a gateway answers a write its computers didn't show with a 504 whose error is its message, so the client saw no
code and write_clock raised the plain error: the always-on step would retry or swallow an applied write, and a hook's
catch would hide it (Greptile). Any 504 whose JSON names its stale computers now counts, whatever its error; a
proxy's own 504 page stays the error it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator Author

Deferring this for now: it adds complexity we don't need yet, and nothing on main depends on it. Without it, core's clock writes don't wait for the computers on the clock, and a computer that falls behind isn't reported. If that turns out to be a problem (for example on Modal, FD-5487), reopen this PR. The branch edgararakelyan/fd-5473-hb13-core-waits-sync_env_clock-post is kept at 3f92b0f.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant