Skip to content

Warn once when a Runtime history exporter starts failing - #972

Merged
SaladDay merged 2 commits into
mainfrom
pstack/runtime-history-export-warning
Nov 19, 2026
Merged

SaladDay merged 2 commits into
mainfrom
pstack/runtime-history-export-warning

Conversation

@SaladDay

@SaladDay SaladDay commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Export errors, timeouts and exporter panics in runtimeobs were discarded without any trace. An unreachable OTLP collector or a failing history insert looked exactly like a Runtime that simply has no history, leaving operators nothing to debug from.

What

  • The export dispatcher logs one warning at the start of each outage, naming the exporter type and outcome class (error / timeout / panic). A recovery followed by a new failure logs again.
  • The error text itself is not logged, since it may carry a collector response.
  • Observation results, the non-blocking queue and drop semantics are unchanged.
  • Docs updated (contracts/agents-api/runtime-observability.md, en + zh).

Verification

  • New exporter_test.go fails on main (want one warning per outage (2), got 0) and passes with this change; also asserts the error text never reaches the log.
  • go test -race -count=20 ./services/core/internal/runtimeobs/ passes; go vet ./... clean.

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Export errors and exporter panics were discarded without any trace, so an
unreachable OTLP collector or a failing history insert looked exactly like
a Runtime with no history. The dispatcher now logs one warning at the start
of each outage with the exporter type and outcome class (error, timeout,
panic); the error text is not logged because it may carry a collector
response. Observation results and queue behaviour are unchanged.
@blacksmith-sh

This comment has been minimized.

@SaladDay
SaladDay merged commit b950f91 into main Nov 19, 2026
25 checks passed
@SaladDay
SaladDay deleted the pstack/runtime-history-export-warning branch November 19, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant