Repository navigation
Feat: Record thinking, effort and beta flags on inference events - #1667
Conversation
Each inference event now states what the request asked for and which thinking blocks travelled with it: - thinkingType and thinkingDisplay, from the request's thinking field - effort, from output_config.effort (Anthropic), reasoning_effort (chat completions) or reasoning.effort (Responses) - anthropicBeta, the anthropic-beta header with every value joined - historyThinking and responseThinking, counts of the thinking and redacted_thinking blocks in the request history and in the response, buffered or streamed: blocks, redacted, textBytes, signatureBytes, redactedBytes The counts show a block that comes back with a signature and no text. That is the shape a LiteLLM gateway at 1.105.1 drops before Bedrock (BerriAI/litellm#41361), and until now the archive had no way to show it. Every new value is decoded as raw JSON and read only when it has the expected type, so a value of another type leaves the request parsed and only that field empty. The fields are short scalars and counts, so the summary view keeps them and the archive carries them in the event record. Part of rossoctl#1494. Signed-off-by: Hai Huang <huang195@gmail.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 40 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
clawgenti
left a comment
clawgenti
left a comment
There was a problem hiding this comment.
Summary
This is clean work. The new fields are all decoded as json.RawMessage and read only when they have the expected type. I ran my own wrong-type probe bodies against head and base, and they give identical results. The bodies cover: object or number thinking text, null thinking/output_config, a numeric thinking.type, stray data/signature on text blocks, wrong-typed buffered response blocks, wrong-typed stream deltas and content_block_start.data, a numeric reasoning_effort, and a list or object reasoning. Messages, completion, usage and finish reason all survive. Streamed thinking counts don't double on a second finalize, because finalize assigns a copy. The buffered OnResponse arm also returns early once the stream has filled Completion. I left one non-blocking suggestion inline.
Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go (pipeline extension, inferenceparser request/buffered/SSE paths, served-model cleaner), tests, docs
Agent/IDE config (.claude/.vscode): none
Commits: 1 commit, all signed-off: yes
CI status: passing (CodeRabbit and Spellcheck skipped)
Local: go test passes for ./plugins/inferenceparser, ./sessionapi, ./session/... and ./pipeline. cost/ledger and cost/usage fail the same DST/tzdata tests on base, so those failures don't come from this PR.
| // agent action. Same classification as the OpenAI path. | ||
| IsAction: true, | ||
|
|
||
| ThinkingType: stringField(req.Thinking, "type"), |
There was a problem hiding this comment.
suggestion: AnthropicBeta goes through cleanServedModel (control runes replaced, capped at 256), but ThinkingType, ThinkingDisplay and Effort are stored exactly as the client sent them. All six fields are kept by view=summary and ride in the archive event record. A long or control-character-laden thinking.type would therefore be stored and served on every summary row, uncapped. Agentop doesn't draw them today, but the next reader that prints them would inherit the escape sequences. The same applies to Effort in parseOpenAIRequest and parseResponsesRequest. Model already has this property, so it isn't new, but for these record-only fields running them through cleanServedModel (or a smaller cap) is cheap. Not blocking.
github-project-automation
Bot
moved this from New/ToDo
to Done
in Rossoctl Issue Prioritization
Summary
Part of #1494. Each inference event now records what the request asked for and which thinking blocks travelled with it:
thinkingType,thinkingDisplaythinking.typeandthinking.displayeffortoutput_config.effort(Anthropic),reasoning_effort(chat completions),reasoning.effort(Responses)anthropicBetaanthropic-betaheader, every value joined with a comma, cleaned and capped asservedModelishistoryThinking,responseThinkingthinkingandredacted_thinkingblocks in the request history and in the response (buffered or streamed):blocks,redacted,textBytes,signatureBytes,redactedBytesWhy
The archive could not show what thinking setting a turn used, or whether its thinking blocks reached the model. A Claude Code session through a LiteLLM gateway at 1.105.1 sends each earlier thinking block back with a signature and empty text, and that LiteLLM version drops such blocks before Bedrock (BerriAI/litellm#41361). With this change, the request row shows it directly, for example
historyThinking: {blocks: 1, signatureBytes: 912}.Design
TestThinkingFieldsOfAnUnexpectedTypeKeepTheRequestParsedsends 9 such bodies. The base commit parses all of them, and so does this one.view=summarykeeps them (the field guard goes from 29 to 35) and the archive carries them in the event record. The interner and the archive format do not change.Not in this PR
reasoning_contentand Responses reasoning items on the response side.Verification
core/plugins/inferenceparser/thinking_test.go, and one archive round trip incore/session/archive/format_test.go.go testpasses for inferenceparser, pipeline, sessionapi, session/... and cost/....go vetandgofmt -lare clean, and agentop builds.thinkingType: adaptive,effort: highand theanthropicBetavalues. The response rows counted the thinking blocks, and the next request counted them in its history with 0 text bytes.Assisted-By: Claude (Anthropic AI) noreply@anthropic.com