Repository navigation
feat(verifiers): read an agent turn's tool calls from any stored trajectory; record every app change whole - #678
Open
sarah-chitty-1 wants to merge 5 commits into
Open
sarah-chitty-1 wants to merge 5 commits into
sarah-chitty-1 wants to merge 5 commits into
Conversation
…ored trajectory; record every app change whole turn_events parses a stored turn trajectory (OTel spans, OpenInference's included; Claude Code and Gemini stream-json) into the agent's text and each tool call with its input and result, dropping what every turn or span repeats (tool lists, the conversation so far, signatures, token counts). readable_result unwraps MCP envelopes, decodes base64 email bodies and leaves images out. The OTel tool-span reading is shared with compact_otel_trajectory, which behaves as before. always_on's stored timeline now keeps every change of a changelog read with its whole summary (no 200-change cap per read, no 200-character cut); the run's metadata record is unchanged. render_for_judge cuts a summary for display only and says when an older run didn't record some changes. Fixtures are redacted recordings of a real always_on wake and a Claude Code turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ol name; a failed Gemini turn keeps its error message Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rded it (Claude Code blocks, a GenAI chat span's thinking blocks) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ks on an LLM span Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 14, 2026
…ssage.contents.N.message_content.text), as GPT models are called Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 14, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
First of three stacked PRs that let an always-on run's process judge read every tool call the assistant made (Linear FD-5840). Today that judge sees only a summary of each wake. This PR adds the two pieces underneath:
judge_utils/turn_events.pyturns a stored turn trajectory into the agent's text and each tool call with its input and result. It reads OTel spans (OpenInference's included, which is what the always-on assistant stores), Claude Code stream-json and Gemini stream-json. It keeps the agent's thinking when the trajectory recorded any (Claude Code thinking blocks, a GenAI chat span's thinking blocks), and drops what every turn or span repeats: the tool list, the whole conversation copied into each model span, thinking signatures, token counts. A tool span named generically (execute_tool) takes its real name from the prompt, and a failed Gemini turn keeps its error message. On a real 109-wake run that's 14.2 MB down to 0.5 MB.readable_resultunwraps MCP result envelopes, decodes base64 email bodies and leaves images out.compact_otel_trajectorykeeps the tool calls on this format but drops the assistant's own text and doesn't pair calls with results. The tool-span reading is now shared between the two (tool_span_io,is_tool_span,extract_final_response);compact_otel_trajectorybehaves exactly as before.render_for_judgenow cuts summaries for display only, and says when an older run didn't record some of its changes.Backward compatibility
compact_otel_trajectoryreturns the same output as before.How it was tested
tst/unit/task_step/verifiers/judge_utils/test_turn_events.py. They run on redacted recordings of a real always-on wake (OpenInference) and a Claude Code turn, plus Gemini stream-json. They check that repeated context is dropped, calls are paired with results, email is decoded, images are left out, and the old compactor is unchanged.test_always_on_who_did_what.py: the cap test is replaced by "every change is kept whole, and the run's metadata stays small".test_turn_events.pyalso covers kept thinking (an empty block dropped), a generically named tool span, and a failed Gemini turn.pytest tst/unit/task_step tst/unit/agents/claude_codepasses with the stack on top (3,372 passed);ruff checkis clean.The re-grade that uses all three PRs is described in #680.
🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no blocking issues remain.Summary
Adds a shared reader for stored turn trajectories and keeps every app change with its full summary.
Reviews (7) · Last reviewed commit: "fix(turn_events): read an OpenInference ..." · Reviewed by Greptile