Skip to content

feat(verifiers): read an agent turn's tool calls from any stored trajectory; record every app change whole - #678

Open
sarah-chitty-1 wants to merge 5 commits into
mainfrom
sarahchitty/fd-5840-turn-reader
Open

sarah-chitty-1 wants to merge 5 commits into
mainfrom
sarahchitty/fd-5840-turn-reader

Conversation

@sarah-chitty-1

@sarah-chitty-1 sarah-chitty-1 commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

First of three stacked PRs that let an always-on run's process judge read every tool call the assistant made (Linear FD-5840). Today that judge sees only a summary of each wake. This PR adds the two pieces underneath:

  • Reading one turn's trajectory, whatever format it's in. judge_utils/turn_events.py turns a stored turn trajectory into the agent's text and each tool call with its input and result. It reads OTel spans (OpenInference's included, which is what the always-on assistant stores), Claude Code stream-json and Gemini stream-json. It keeps the agent's thinking when the trajectory recorded any (Claude Code thinking blocks, a GenAI chat span's thinking blocks), and drops what every turn or span repeats: the tool list, the whole conversation copied into each model span, thinking signatures, token counts. A tool span named generically (execute_tool) takes its real name from the prompt, and a failed Gemini turn keeps its error message. On a real 109-wake run that's 14.2 MB down to 0.5 MB. readable_result unwraps MCP result envelopes, decodes base64 email bodies and leaves images out.
    • The existing compact_otel_trajectory keeps the tool calls on this format but drops the assistant's own text and doesn't pair calls with results. The tool-span reading is now shared between the two (tool_span_io, is_tool_span, extract_final_response); compact_otel_trajectory behaves exactly as before.
  • Recording every app change in full. The always_on step's stored timeline now keeps every app change, each with its whole summary. Before, it kept at most 200 changes per changelog read and cut each summary at 200 characters; real runs make up to about 3,600 changes. This only affects the timeline object in the object store; the run's metadata record still holds counts only. render_for_judge now cuts summaries for display only, and says when an older run didn't record some of its changes.

Backward compatibility

  • No task setting changes.
  • compact_otel_trajectory returns the same output as before.
  • The judge's wake-by-wake prompt text is the same as before for runs that kept every change; summaries are still cut at 200 characters for display.

How it was tested

  • New tests: tst/unit/task_step/verifiers/judge_utils/test_turn_events.py. They run on redacted recordings of a real always-on wake (OpenInference) and a Claude Code turn, plus Gemini stream-json. They check that repeated context is dropped, calls are paired with results, email is decoded, images are left out, and the old compactor is unchanged.
  • test_always_on_who_did_what.py: the cap test is replaced by "every change is kept whole, and the run's metadata stays small".
  • test_turn_events.py also covers kept thinking (an empty block dropped), a generically named tool span, and a failed Gemini turn.
  • pytest tst/unit/task_step tst/unit/agents/claude_code passes with the stack on top (3,372 passed); ruff check is clean.

The re-grade that uses all three PRs is described in #680.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no blocking issues remain.

Summary

Adds a shared reader for stored turn trajectories and keeps every app change with its full summary.

  • Always-on runs keep every app change with its full summary.
  • Stored agent turns become ordered text and tool events.

Reviews (7) · Last reviewed commit: "fix(turn_events): read an OpenInference ..." · Reviewed by Greptile

…ored trajectory; record every app change whole

turn_events parses a stored turn trajectory (OTel spans, OpenInference's included;
Claude Code and Gemini stream-json) into the agent's text and each tool call with its
input and result, dropping what every turn or span repeats (tool lists, the
conversation so far, signatures, token counts). readable_result unwraps MCP
envelopes, decodes base64 email bodies and leaves images out. The OTel tool-span
reading is shared with compact_otel_trajectory, which behaves as before.

always_on's stored timeline now keeps every change of a changelog read with its
whole summary (no 200-change cap per read, no 200-character cut); the run's
metadata record is unchanged. render_for_judge cuts a summary for display only and
says when an older run didn't record some changes.

Fixtures are redacted recordings of a real always_on wake and a Claude Code turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 2 commits November 13, 2026 18:28
…ol name; a failed Gemini turn keeps its error message

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rded it (Claude Code blocks, a GenAI chat span's thinking blocks)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ks on an LLM span

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssage.contents.N.message_content.text), as GPT models are called

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant