Skip to content

Diagnose slow worker shutdown in warning test - #2098

Draft
VegetarianOrc wants to merge 1 commit into
mainfrom
diagnose-worker-shutdown-timeouts
Draft

VegetarianOrc wants to merge 1 commit into
mainfrom
diagnose-worker-shutdown-timeouts

Conversation

@VegetarianOrc

@VegetarianOrc VegetarianOrc commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Keep the original worker lifecycle in test_warns_when_workers_too_low, but log the task queue and pending asyncio task stacks at 5, 25, and 55 seconds when shutdown is slow. The warning test has intermittently timed out at 60 seconds in CI without identifying which operation remained pending. This draft is for collecting CI evidence, not for changing worker behavior or landing a fix.

Local tracing of the unmodified test saw one activity poll remain pending for 53.947 seconds while the workflow poll finished in 0.002 seconds; the first worker finished in 58.3 seconds. Core relies on server-driven poll completion when the server advertises that capability. The new diagnostics can identify the pending Python/bridge task and correlate the task queue with Core/server logs, but cannot alone determine the ShutdownWorker RPC result.

Validation

  • poe test -s -k 'test_warns_when_workers_too_low or test_can_run_resource_based_worker or test_can_run_composite_tuner_worker' tests/worker/test_worker.py: 3 passed.
  • uv run pyright tests/worker/test_worker.py, Ruff import/format checks, and git diff --check: passed.
  • Deliberately delayed an activity poll using a temporary local pytest plugin and confirmed the 5-second diagnostic logs the pending activity poll. This plugin is not part of the PR.

No broader test suite or full lint was run for this diagnostic-only change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant