[sarah-chitty-1]

What and why

Third of three stacked PRs (Linear FD-5840); it builds on #679. The always-on process judge can now read every tool call of every wake.

The problem.

• Today always_on_agent_verifier grades from the user's chat and a summary of each wake, and refuses use_trajectory.
• Agent judges on other long runs were given each turn as one line of up to 1.8 MB. The judge's Read tool refuses lines that long, so it wrote Python to read them, and timeouts were the most common failure.
• On Claude Code trajectories the user's replies were missing entirely.

What this PR adds.

• always_on_staging.py builds the run's files from its stored timeline and each wake's trajectory. Each trajectory is matched to its wake by the timeline's trajectory_url, never by list position. The files:
  • wakes/wake_NNN.jsonl, one event per line. First what woke the assistant, with the user's whole message. Then every tool call with its input and result, and the assistant's own text. Then the rest of the wake in time order: every app change, its closing reply, messages to the user, nudges, and the user's reads.
  • events.jsonl: every wake in one file.
  • triggers.jsonl: every trigger firing.
  • index.md: a block per wake with its time, what woke it, the user's message, its write actions and messages (each with file:line), and errors.
  • Any line too long for the Read tool keeps a preview and moves its long fields to wakes/results/. Lines longer than 1,800 characters there are split across lines, nothing dropped (a JSON file that needed it becomes .txt).
  • The agent's thinking, when its trajectory recorded any, is an assistant_thinking line in its wake.
  • A write action is a call of a tool that made one of that wake's app changes, from the apps' change logs. If the apps keep no change log, it falls back to a guess from the tool name, and index.md says which method it used. In a run with change logs, a write-sounding call of a tool no change names (an app without a log) is listed as a "possible action".
  • triggers.jsonl places each firing after the right wake by comparing parsed times; the staging log gives a count of untrimmed trajectories, not the list.
  • Missing or unreadable trajectories, and turns compaction didn't shrink, are recorded and logged.
• always_on_agent_verifier:
  • Accepts use_trajectory with the agent judge only.
  • Writes the files into the judge's sandbox concurrently.
  • Adds a prompt note on reading them: start at index.md, search, then read with an offset; later actions replace earlier ones; where the user's messages are; cite wake and call numbers; what counts as the assistant's own act.
  • Records the staging and what each judge call read (judge_reads, counting files read through Read, Grep or Bash).
  • Gives the judge only Bash and Read (builtin_tools: {"only": ["Bash", "Read"]}) and no peer-agents tools (peer_tools: false): no helper agents, no web, no writes, no messaging other agents. Its CLI command is --tools Bash,Read --strict-mcp-config with no --mcp-config.
  • Defaults: 2,400 s and 400 turns per judge call, one timeout retry, at most 3 batches at once. Batch size still comes from the task.
  • Without a stored timeline, it falls back to rubrics_verifier's usual per-turn loading.

Re-grade of a real run

Silent Week (psi-ts-rw-silent-week-always-on-ab4d9621782ac6f6: 109 wakes, 3,586 app changes) was re-graded locally. The data was read-only from the hub. The judge was the Claude Code CLI on Opus 5.5 with 1M context, run with the agent's own command builder. The criteria were ML's 8 original process criteria.

• Cost and time: 36 s, $0.71, one judge call, 8 turns, no timeouts or retries.
• How it read the files:
  • It searched index.md, made one Python pass over every wake's tool calls, then opened wakes 9, 11, 18, 22, 59, 60, 71, 72, 75 and 80.
  • It didn't open events.jsonl or triggers.jsonl.
  • The judge's tool list had no Agent, Workflow or Monitor.
• Verdicts: all 8 criteria passed, each with reasons citing wakes.
• Spot-checks: two reasons were checked against the staged files and are right: W23 ("Mei Sato has approved the endoscopy") and W60 ("never send words as her").

Then the full path: the judge deployed the normal way (the Claude Code agent in local Docker, files staged into its sandbox, settings sent through agent-config and read back), ML's original criteria:

Run	Result	Cost
Silent Week (109 wakes)	8/8 passed, agreeing with the re-grade above	$0.61–0.69
Holiday Rush (psi-ts-craftweb-holiday-rush-always-on-0dc41c6159628433), 4 batches of 4, 3 at once	9/15; the 6 fails are real (no forecast, extra pour days never asked, 4 boxes of wax, the Rae offer, "DONE" with Rae unanswered)	$0.81
Fender Bender (psi-ts-rw-fender-bender-always-on-ef40b3a3fb2082d1, cut off at wake 80)	2/6; real fails (no footage, no rental, no repairs, agreed to a recorded statement)	$0.49
Silent Week with a 20 s limit and 1 retry	Timed out, retried once, both judge trajectories kept, failed with JudgeTimeoutError	~$0

The read-back config showed max_turns: 400 and timeout_seconds: 2400. Those runs exposed peer-agents, web and write tools on the judge, which this PR now removes (above).

Backward compatibility

• An always_on_agent_verifier without use_trajectory grades from the same prompt as before, with these default changes:
  • 2,400 s and 400 turns per judge call;
  • one timeout retry;
  • at most 3 batches at once;
  • the judge gets only Bash and Read, and no peer-agents tools.
• A task that sets these explicitly keeps its own values.
• The converted always-on tasks store judge_timeout_seconds: 1000 and use_agent_judge: false, so they grade as today until their task files turn this on.

How it was tested

• test_always_on_staging.py, on a redacted recorded timeline and recorded trajectories. It covers:
  • turns matched to wakes by URL;
  • a nudge in its wake;
  • missing and unreadable trajectories flagged;
  • the user's message first;
  • 250 app changes kept whole;
  • long results and long lines moved to side files;
  • decoded email;
  • index.md's file:line pointing at the right line;
  • write actions from the change logs, and the name-guess fallback;
  • events.jsonl and triggers.jsonl;
  • 120 wakes sorted in order;
  • an unknown format flagged;
  • judge_reads across Read, Grep, Bash loops and globs.
• test_always_on_agent_verifier.py covers an end-to-end grade staged into a fake judge sandbox, the fallback without a timeline, a run that turns the agent judge off being refused, and the defaults.
• tst/unit/agents/claude_code/test_always_on_judge_config.py checks that the judge's settings are ones the Claude Code agent's card takes.
• pytest tst/unit/task_step tst/unit/agents/claude_code passes (3,372 passed); ruff check is clean.
• Full tst/unit: 3 failures, in test_disk_recipe and test_docker_image_github_clone. They also fail on main.

🤖 Generated with Claude Code

Confidence Score: 4/5

The PR should not merge until Codex and OpenCode judges can accept the tool settings.

Findings

1.  Other judges cannot grade ▶

Fix with agent prompt

### Issue 1
src/agent_env/task_step/task_steps/verifiers/always_on_agent_verifier.py:181-182
`judge_agent_config` sends Claude Code's tool names to every agent judge. If a task selects Codex or OpenCode through `judge_a2a_agent_id` or `agent_name`, those agents accept `builtin_tools` but reject `Bash` and `Read`. Codex uses names such as `exec_command`; OpenCode uses lowercase `bash` and `read`.

The config request fails before grading starts, even without `use_trajectory`. Choose the tool selection from the judge's supported names instead of sending Claude Code names unconditionally.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds searchable files for grading always-on runs with use_trajectory.

• Always-on judges can search every wake’s tool calls as files.
• Always-on judges get longer limits and fewer tools by default.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Timeline[Stored timeline] --> Stage[Match trajectories to wakes]
  Trajectories[Wake trajectories] --> Stage
  Firings[Trigger firings] --> Stage
  Stage --> Files[Index and staged files]
  Files --> Judge[Agent judge]
  Judge --> Grade[Criterion results]
  Judge --> Reads[Recorded file reads]

Loading

_{Reviews (10) · Last reviewed commit: "Merge #679 into #680" · Reviewed by Greptile}

[greptile-apps]

Other judges cannot grade

judge_agent_config sends Claude Code's tool names to every agent judge. If a task selects Codex or OpenCode through judge_a2a_agent_id or agent_name, those agents accept builtin_tools but reject Bash and Read. Codex uses names such as exec_command; OpenCode uses lowercase bash and read.

The config request fails before grading starts, even without use_trajectory. Choose the tool selection from the judge's supported names instead of sending Claude Code names unconditionally.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/verifiers/always_on_agent_verifier.py
Line: 180-181

Comment:
**Other judges cannot grade**

`judge_agent_config` sends Claude Code's tool names to every agent judge. If a task selects Codex or OpenCode through `judge_a2a_agent_id` or `agent_name`, those agents accept `builtin_tools` but reject `Bash` and `Read`. Codex uses names such as `exec_command`; OpenCode uses lowercase `bash` and `read`.

The config request fails before grading starts, even without `use_trajectory`. Choose the tool selection from the judge's supported names instead of sending Claude Code names unconditionally.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.