[sarah-chitty-1]

Stacked on #681 (files in the opening, the rating step, the two new persona sections); merge that first.

A new verifier step, user_sim_judge, grades the user sim of a run (the AI playing the user), not the assistant (FD-5490). It covers always_on runs and prompt_agent chats with the built-in user sim (deploy_user_sim_agent).

It only records verdicts for now. Nothing marks a run or changes what graders read until the judge is calibrated against a person's labels. Marking is behind mark_user_sim_error, off by default.

What it does

• Rebuilds the user sim's exact instructions from its recorded settings, and checks them against the recorded prompt_sha256.
• Reads every check-in the user sim got and every answer it gave: always_on's stored timeline, or the chat's user_sim_reads entries.
• Asks one narrow question per check, mapped to Anisha's User Simulator Rubric Taxonomy (PSI Bench SSOT):

Check	Families
guidance: gave the requirements and decisions the assistant needed	A1, A3
oversharing: leaked look-up facts, private details or credentials	G1, G2
delegation: permission given or withheld per its rules; asks once about a missing "tell you after" item	A4
style: sounds like the person, reacts in character, fits its moment	F1–F4
rogue: acted like the assistant, broke character, let the agent off	G3, G5
made_up: invented facts or contradicted itself	A2
ending: said done at the right time	G4

• A task can add its own rubric_items under any family, in the shape of a criterion in a task's usersim.yaml (psi-benchmark-tasks turns one into the other). Each item rides along with the check that owns its family, at most 12 to a call:

rubric_persona_sha256: 3f1c...   # all 64 hex digits
rubric_items:
- {id: a1_call_rae, family: A1, kind: hard, name: "Rae: yes if it fits",
   check: "...", na_when: "The assistant didn't raise Rae's order."}
- {id: f2_her_length, family: F2, kind: auto, check: "Every reply is at most 25 words.", metric: max_words, max: 25}

• kind: hard (an exact rule from the user sim's instructions, failed on any breach), general (judged), or auto (counted, not judged).
• na_when: when the item never came up. always makes it n/a with no call and needs a static_na_reason; never means it always applies. An auto item takes only always.
• Task items are tied to the persona, not the task version. rubric_persona_sha256 is the fingerprint of the persona and opening the items were written for (persona_fingerprint, which deploy_user_sim_agent records as the run's persona_sha256). If they differ, the task items aren't graded and the entry says why; the general checks still run. Other task edits don't affect it.
• Satisfaction (A5) is the separate rate_user_satisfaction step (feat(user sim): files in the first message, and a rating step after the run #681).

Settings and output

• One judge model, judge_model, defaulting to the same model rubrics_verifier uses. A run-wide --judge-model doesn't change it; only this step's own step_params do.
• Each check writes its reason before its verdict. A fail that cites no graded turn, or whose reason ends by saying no failure was found, is asked again; a second bad answer is unknown.
• Verdicts, reasons and cited turns go to metadata["user_sim_judge"][<verifier id>]. There's no verifications entry, so nothing mixes into the assistant's scores.
• A run that ended in an error is skipped. A broken judge never fails the task (fail_task_on_error false).
• The run's verdict is yes / no / unknown. Unknown means a check that could mark the run kept failing to answer; only yes marks, when marking is on. Style never marks a run on its own.
• If the user sim's replies reach its max_replies, the judge is told the reply limit, which the user sim itself never sees, so the ending check can allow a short sign-off there.
• Cost: 7 calls per run, roughly $0.08 for a short chat and $0.24 for a multi-day run on Sonnet, uncached.

Tests

tst/unit/task_step/verifiers/test_user_sim_judge.py (28) uses a fake model, with one always_on and one chat run. tst/unit/task_step and task/episode: 3,217 passed; the two test_mcp_cli_builder tests that need click were deselected. No paid calls.

Not done: calibration. The old test labels are AI-written, so the judge will be calibrated on a person's own labels of about 60 answers before marking is turned on.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge after its stated dependency, PR #681.

Summary

Adds a record-only judge for the user simulator, separate from the assistant’s scores.

• Runs can now record a separate grade of the simulated user.

_{Reviews (12) · Last reviewed commit: "user_sim_judge: an auto item can't pass ..." · Reviewed by Greptile}