[sarah-chitty-1]

What and why

Second of three stacked PRs for always-on grading (Linear FD-5840); it builds on #678. It makes small, opt-in additions to rubrics_verifier that the always-on verifier in #680 uses. Every one is off unless a task or subclass turns it on.

• judge_timeout_retries (unset by default). When an agent judge call runs out of time, it is tried again. The timed-out attempt's trajectory is fetched and logged in judge_output_discrepancies (kind: judge_timeout), so there's a record of what the judge did. If the last attempt also runs out, the error (JudgeTimeoutError) carries that trajectory's link. A batch retries on its own, so the other batches keep going.
  • "Ran out of time" means the poll's own deadline, or a failed task whose error code says so (<harness>.timeout, as OpenHands and pydantic-ai send).
  • When the code doesn't say (none, or Claude Code's cli_error), it falls back to the message phrase "timed out after Ns", which Claude Code already sends.
  • A model API timeout ("The operation timed out") is not treated as the judge running out of time.
• judge_max_turns (unset by default): sent to the judge as max_turns.
• The Claude Code agent's peer_tools setting (default true, so every other use is unchanged). False leaves the SDK's peer_agents MCP server out of the turn, so the agent can't message or start other agents. The card lists it.
• stage_trajectory, an override point. A subclass can stage the run in the judge's sandbox itself and say where (StagedTrajectory). Returning None, the default, loads the trajectory exactly as before.
• judge_agent_config, an override point for extra judge settings; {} by default. It logs any setting the judge's card doesn't take.
• Run.firings_read(): the trigger firings of every log that could be read.

Backward compatibility

• With none of the new settings, rubrics_verifier behaves as before: a timeout still fails at once, nothing extra is fetched, and the judge config, prompt and trajectory loading are unchanged.
• The Claude Code agent is unchanged unless a caller sets peer_tools: false. Its error codes and messages are as before.
• One fix that applies to every verifier: if loading the trajectory into the judge's sandbox fails, a judge the verifier deployed itself is now terminated (before, it was only terminated after grading). The existing termination code moved into one helper, _terminate_auto_deployed_judge, used by both paths.

How it was tested

• New: tst/unit/task_step/verifiers/test_rubrics_verifier_judge_timeout_retry.py. It covers:
  • the default failing at once;
  • a retry that keeps the timed-out trajectory;
  • a harness's timeout code and the message fallback (Claude Code's cli_error with "timed out after Ns");
  • three failures that are not retried;
  • round-trips and validation;
  • extra config sent only when the judge's card takes it.
• A Claude Code agent test checks that peer_tools: false leaves the peer_agents server out of the turn.
• New: test_rubrics_verifier_staging_cleanup.py checks that a failed staging terminates the judge the verifier deployed (it fails without the fix).
• pytest tst/unit/agents/claude_code tst/unit/task_step/verifiers passes at this commit (771 passed), and tst/unit/task_step passes with feat(always_on_agent_verifier): with use_trajectory, the agent judge searches every wake staged as files #680 on top (3,372 passed); ruff check is clean.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no actionable issues remain.

Summary

Adds optional judge timeout retries, a turn cap, staging and config hooks, and a Claude Code peer_tools setting.

• Agent judges can retry calls that hit their time limit.
• Verifier subclasses can set judge turn limits and options.
• Subclasses can stage the run where an agent judge reads it.
• Failed trajectory staging now releases a verifier-deployed judge.
• Claude Code turns can leave peer-agent tools out.
• Run callers can read trigger firings from available logs.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Prepare judge] --> B[Stage trajectory]
    B -->|Staging fails| C[Terminate auto-deployed judge]
    B -->|Ready| D[Run judge]
    D -->|Timeout retries enabled| E[Capture trajectory]
    E --> F{Retries remain?}
    F -->|Yes| D
    F -->|No| G[Raise timeout with trajectory link]
    D -->|Completed| H[Check judge output]
    H --> I[Terminate auto-deployed judge]
    G --> I

Loading

_{Reviews (8) · Last reviewed commit: "Merge #678 into #679" · Reviewed by Greptile}