Run the agentic eval through the Harbor workflow hosted in lightcone-evals by aboucaud · Pull Request #255 · LightconeResearch/lightcone-cli

aboucaud · 2026-10-22T06:47:43Z

Acknowledgements

The eval this PR calls is built on lightcone-bench:

@dkn16 (Kangning) designed and built lightcone-bench. The eval takes these from it:
- the cross-harness Harbor runs;
- the oracle gate;
- skipping a leg whose secret is missing;
- the ARG LIGHTCONE_COMMIT= stack pin.
The snae-build task is his port of this repository's earlier eval.
@cailmdaley (Cail) contributed the trajectory metrics (lightcone-bench#1). The agent-step count in the eval report is his metric.

What

The agentic eval now runs on Harbor through a reusable workflow hosted in lightcone-evals, beside the task it runs (LightconeResearch/lightcone-evals#5, merged). This repository keeps only a caller:

.github/workflows/eval.yml (about 25 lines) runs on non-draft PRs, pushes to main and dispatch. It calls LightconeResearch/lightcone-evals/.github/workflows/eval.yml@main, and passes this repository's commit as stack_ref and its own secrets through secrets: inherit.
- Per leg, the hosted workflow runs the task's oracle, which must score 1.0, then the claude-code agent on claude-sonnet-5-5 with the agent-skills main skills installed through --skill.
- The report shows the verifier's reward, agent steps and skills loaded, runs harbor analyze with the task's rubric, and posts a sticky comment.
- It fails on any of: a failed oracle, a reward below 1.0, or an errored trial or analysis. Agent steps are reported, not gated.
Removed: the old eval (.github/scripts/trace_digest.py, evals/prompt.md, evals/tasks/, evals/trace-analysis-prompt.md). The task lives in lightcone-evals.
CLAUDE.md: the eval paragraph is now a pointer.

Decided

The agent-skills version is main (discussion, decision).
The workflow moves to lightcone-evals. Moving it was decided on this PR; it now lives next to the task contract it reads (the stack pin, reward.json keys, rubric.toml). lightcone-evals is public, so no token is needed, and a caller's secrets stay its own.
Francois's slimmer layout:
- --skill from a git source;
- the oracle runs in each leg's job;
- reward and steps are read from reward.json.
No step gate for now, and no comparison with main. The eval keeps no main history, so the caller drops actions: read.
The task instruction says the run is unattended (snae-build: say the run is unattended lightcone-evals#4, merged). Before that, the agent stopped to ask a question.

Verified

The hosted workflow on hosted-eval, before Add prism setup command, replace prism remote #5 merged (4f48c24):
- oracle 1.0;
- claude-code reward 1.0 in 11 agent steps, with skills loaded (astra@278ecdb, lightcone@278ecdb);
- all four rubric criteria pass.
The caller now points at @main (6919e8b). This PR's next eval run is the first against lightcone-evals main.
actionlint is clean on the caller.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

The eval now runs evals/snae-build, a Harbor task inspired from lightcone-evals' port of this repository's eval, in one workflow: prepare (the engine wheel built from the checkout into the task image, the agent-skills bundles), an oracle gate, claude-code and codex legs with the skills installed through Harbor's --skill, and a report. The instruction is now the one line a user would type, so how to drive lc and astra has to come from the skills and the CLIs' help. The report gates on reward, breakage and mean agent steps against the median of the leg's last main runs, and always ships a trace analysis that names where the steps went and which change most plausibly caused it. agent-skills reuses the workflow through workflow_call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

aboucaud · 2026-10-22T06:52:35Z

test (ubuntu-latest, 3.13) is red for a reason outside this PR.

All 7 failures are in tests/test_container_smoke.py. Each image build stops at:

RUN UV_PYTHON_INSTALL_DIR=/opt/python uv python install 3.13.17
error: No download found for request: cpython-3.13.17-linux-x86_64-gnu

The runner's interpreter is now CPython 3.13.17. That is the .python-version lc init writes, and so the version the image installs. The image's pinned uv (image.UV_IMAGE, ghcr.io/astral-sh/uv:0.12.23) predates 3.13.17 and has no download for it. main was last green on Oct 15, before the runner moved, and would fail the same way today. This PR touches no src/ or tests/ file.

No open PR fixes it. The fix is the #248 shape: bump UV_IMAGE in src/lightcone/engine/image.py to a uv that knows 3.13.17 (the current release, uv 0.13.5, lists cpython-3.13.17-linux-x86_64-gnu), with its manifest-list digest. That moves the identity document, so containerized outputs read behind. That is an engine change of its own, so I'm proposing it as a separate PR rather than folding it in here. A re-run would hit the same missing download, so I haven't triggered one.

Generated by Claude Code

github-actions · 2026-10-22T06:54:57Z

❌ Eval

lightcone-evals 8dce552 · lightcone-cli/snae-build

Leg	Oracle	Reward	Agent steps	Skills loaded	Skills injected	lightcone-cli
claude-code	❌	—	—	—		`1342372`

❌ claude-code: reference solution failed — the task or the stack is broken

Trajectories are in the eval-leg-* artifacts.

Run: https://github.com/LightconeResearch/lightcone-cli/actions/runs/41646019694

The tasks stay in LightconeResearch/lightcone-evals, in Harbor format. prepare fetches the requested set (a comma-separated `tasks` input, a folder meaning every task in it, at `tasks_ref`) and rewrites each task's `ARG LIGHTCONE_COMMIT=` line to the lightcone-cli commit under test — the contract lightcone-bench already uses — so no wheel is built and the same workflow serves agent-skills and lightcone-evals as callers. The report groups by leg and task, so steps, history and the gate are per task, and the analyst is handed the task's own instruction. lightcone-evals is private, so a REPO_READ_TOKEN secret is required. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

aboucaud · 2026-10-22T07:33:06Z

Eval / prepare is red: the REPO_READ_TOKEN secret is missing.

Since d27a8a5 the eval fetches its tasks from LightconeResearch/lightcone-evals. That repository is private, so the workflow needs a token that can read it. The step fails on purpose, with REPO_READ_TOKEN is not set — it needs contents:read on LightconeResearch/lightcone-evals.

Fix: add a repository secret REPO_READ_TOKEN here (Settings → Secrets and variables → Actions). It should be a fine-grained PAT, or an org GitHub App token, with Contents: read on LightconeResearch/lightcone-evals. lightcone-bench uses the same secret name. agent-skills will need the same secret once it calls this workflow.

Making lightcone-evals public would remove the need for the token. Nothing in this PR can supply the token, so I'll re-run once it's in place.

Generated by Claude Code

This fixes the `test_container_smoke.py` failures that every branch now hits. #255 shows them, for example. ## What broke GitHub's runner Python moved to **CPython 3.13.17**. `lc init` writes the running interpreter's exact patch to `.python-version`, and the image installs that interpreter with its pinned uv. uv 0.12.23 has no download for 3.13.17, so every image build stops at: ``` RUN UV_PYTHON_INSTALL_DIR=/opt/python uv python install 3.13.17 error: No download found for request: cpython-3.13.17-linux-x86_64-gnu ``` `main` was last green on Oct 15, before the runner moved. ## The fix `image.UV_IMAGE` moves to `ghcr.io/astral-sh/uv:0.13.5`, pinned by its manifest-list digest. This is the same change as #248. - **Why 0.13.5:** it is the first uv release with a 3.13.17 download. I checked `uv python list 3.13.17 --all-versions` against 0.12.24 and 0.13.4, and neither has one. - **Digest:** fetched from ghcr's manifest endpoint for the tag. The same query for `0.12.23` returns the digest currently in `image.py`. - **0.13.0 breaking changes:** none of them touch what the image uses uv for. The image installs a pinned interpreter and runs `uv sync`. The default-Python change does not apply, because the version is always explicit. ## Consequence The uv pin is part of the image identity document, so containerized outputs read `behind` after this, as with any uv bump. Direct-mode projects do not move. ## Tests - `test_image.py`, `test_container.py` and `test_identity.py` pass (96). - `ruff` and `mypy` are clean. - The container smoke suite can't build images in my sandbox, which has no network access for builds, so this PR's CI is its real run. --- _Generated by [Claude Code](https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4)_ Co-authored-by: Claude <noreply@anthropic.com>

aboucaud · 2026-10-22T09:46:33Z

Which agent-skills version should the eval install?

This is open for discussion; nothing in the PR changes because of it yet.

Today: prepare installs agent-skills at skills_ref, which defaults to main. It copies the built plugins/*/skills/*/ bundles, the same ones the marketplace serves. An agent-skills PR (as a workflow_call caller) always uses its own head commit. The exact commit is recorded in refs.json and in the report header (agent-skills@278ecdb on this PR's first run).

What "published" means for agent-skills

Both marketplace manifests point at local paths ("source": "./plugins/lightcone"). claude plugin marketplace add LightconeResearch/agent-skills therefore installs whatever the default branch holds. There is no separate release artifact.
The plugin version in skills.config.json (0.0.6 now) is what tells an existing install to update. It doesn't point to a copy of the files.
There are no release tags to pin to. The only tag, v0.0.2, predates the lightcone plugin.

Today main (278ecdb) is the 0.0.6 release commit, so main and "latest published" coincide. They split as soon as a skill change merges without a version bump. It is then live for new installs, while existing users don't see it until the next bump.

The weakness of following `main` as it moves

A lightcone-cli PR's agent steps are compared with the median of recent lightcone-cli main runs. Each of those runs, and the PR run, used whatever agent-skills main was at the time. A skill rewrite merged in between would then show up as a step change blamed on the lightcone-cli PR.

Options

	Skills used by lightcone-cli runs	For	Against
A. `main` (current)	Newest agent-skills commit	Matches a fresh install	Skill changes move the lightcone-cli baseline, and the PR gets blamed
B. Last release	The commit that set the current plugin `version`	Matches what existing users run; skill changes reach the eval only when released	Needs agent-skills to tag releases (e.g. `lightcone-v0.0.6` in the release PR), or the workflow to search history for the bump
C. The baseline's commit	The agent-skills commit the latest successful lightcone-cli `main` run used, read from its `eval-results`	PR and baseline always run identical skills, so a difference is the lightcone-cli change; small, all in this repo	Skill updates reach lightcone-cli PRs only after the next lightcone-cli `main` run

None of these affect an agent-skills PR, which always runs its own head.

Proposal

C for PR runs, with main runs choosing the skills. Under the proposal, main runs use agent-skills main today. Once agent-skills tags each plugin release, they would use the latest release tag (B), so the baseline tracks what users have rather than unreleased edits.

The step history should also start fresh, or be marked, whenever the skills commit changes between main runs. Otherwise a release that legitimately changes step counts either fails PRs or gets absorbed slowly by the median.

The lightcone-cli side is the prepare step plus a few lines in eval_report.py. The tagging would live in agent-skills' release process.

Generated by Claude Code

The report script loses its heuristics: the phase classifier, the failure regex (it flagged every `sandbox:` header line as a failure), repeated-command counts and the per-task grouping. The trace analyst reads the digest and is handed only the exact facts: reward, steps, harness exceptions and the verifier's output. The step gate waits for three main runs and needs both 25% and three steps over the median, so one noisy trial cannot fail a PR. An oracle failure is now reported as one, rather than as every leg skipped. The workflow runs one task, drops lightcone-evals as a caller and the attempts/tolerance/tasks inputs, and fetches history only for the repository under test. The cost column goes: under OAuth it is an API price nobody pays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

prepare copied eval_report.py and the analysis prompt out of the lightcone-cli checkout at stack_ref, while the workflow ran from the ref its caller pinned — so a stack_ref older than the script broke the run, and agent-skills could get a script that did not match its workflow. eval.yml is now one self-contained file. The numbers and the gate are jq over each trial's result.json and ATIF trajectory, as the original eval did. The trace digest is gone: the analyst reads the raw trajectories (this run's and main's) itself, with Read and Grep only, and the prompt is inline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

aboucaud · 2026-10-22T16:39:03Z

@EiffL for the agent-skill question above I would go with option C which ensures that a PR eval check is doing its job by comparing results with what's on main

Do you agree ?

aboucaud · 2026-10-22T16:40:53Z

+# agent-skills reuses it, with its own PR as the side under test:
+#
+#   jobs:
+#     eval:
+#       uses: LightconeResearch/lightcone-cli/.github/workflows/eval.yml@main
+#       secrets: inherit


This is what it allows now that everything has been folded in a single workflow.

Callers choose their harness legs as a JSON list of Harbor agents, each naming the secret it needs and optional extra variables; lightcone-cli runs one claude-code leg by default. require_skills fails a leg that never loaded a skill, for agent-skills' checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

EiffL · 2026-10-22T19:33:47Z

@aboucaud let's do the simplest, I think testing against main is very ok, so option A. On agent-skills we never modify main apart from when we merge a PR that ticks the version of the skills at the same time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

aboucaud · 2026-10-23T08:21:37Z

Eval report is red. The workflow itself worked end to end (oracle, claude-code leg, analysis). The red comes from the agent's behaviour, not from a defect in this PR.

What happened:

The agent stopped after 5 working steps to ask whether error_model: stat_and_sys can be honoured with the data it was given, which scored reward 0.25. The eval report above has the details.
The task's instruction in lightcone-evals is one line and never says the run is unattended. The lightcone skill tells the agent to work with the user. So asking was a reasonable choice, and in a headless run nobody answers.
The earlier green run (1db117b) used the vendored prompt, which did say the run was automated.

Proposed fix, in the task rather than here: one sentence in lightcone-cli/snae-build/instruction.md, along the lines of "Nobody is available to answer questions: where something is ambiguous, make a reasonable choice, record it, and finish." It states a fact about the environment and gives no hint about how to drive lc.

Because the change touches the shared task that lightcone-bench also runs, I'm waiting for a decision before pushing it. I haven't re-run: one more try wouldn't change the cause.

Generated by Claude Code

The workflow moved beside the task it runs, so this repository keeps a caller that passes the commit under test and its secrets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

aboucaud · 2026-10-23T16:57:34Z

@EiffL, a heads-up: this PR has changed shape since you were asked to review it. The eval workflow moved to LightconeResearch/lightcone-evals#5, beside the task it runs, and folds in your simplifications:

--skill from a git source;
the oracle in the agent job;
reading reward.json directly.

harbor analyze with the task's rubric, the comparison with main and the step gate stay.

Here, what's left is a 26-line caller plus the removal of the old eval. The review that matters now is lightcone-evals#5, and the description above has the merge order.

Generated by Claude Code

The hosted workflow no longer reads main's history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

aboucaud · 2026-10-23T17:32:14Z

claude-review on 4f48c24 ended in 0.5 s at zero cost with is_error: true. That isn't this diff. The Claude token was rate-limited at the same time: agent-skills#56's eval leg died with ApiRateLimitError minutes earlier, and its claude-review failed the same way. I'll re-run it once the limit has had time to reset.

Generated by Claude Code

LightconeResearch/lightcone-evals#5 is merged, so the caller follows its main branch for both the workflow and the task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

Keep main's ui/ structure line and drop the evals/ line, which this branch removes along with the old eval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

aboucaud · 2026-11-03T13:41:15Z

The eval leg on 1342372 fails before any agent runs: the oracle errors with a RuntimeError after 20 s.

Cause: #262 (lc view, now on main) added a hatch build hook. It runs npm ci && npm run build for any non-editable build from a checkout. The snae-build image installs lc with uv tool install git+…@<sha>, which is exactly that kind of build, and the image has no Node. The image build therefore fails.

This isn't this PR's diff. It came in with the merge of main, and any commit after Add lc view: the analysis as one live page, served by Jupyter #262 hits it.
Locally, the same uv tool install git+…@1342372 succeeds once Node 22 is on PATH.

Fix: node-for-lc-build in lightcone-evals (0449b55) copies Node 22 from node:22-slim into the image for the install, then removes it. That is the Node version this repository's CI uses for the front end. The self-test is dispatched on that branch. Its oracle runs against lightcone-cli main, so it exercises the hook. Once that change is on lightcone-evals main, this PR's eval needs one re-run, because the caller pins @main.

Generated by Claude Code

aboucaud mentioned this pull request Oct 22, 2026

Bump the image's pinned uv to 0.13.5 #256

Merged

Merge remote-tracking branch 'origin/main' into eval-harbor

b607c8e

claude added 2 commits October 22, 2026 13:11

aboucaud commented Oct 22, 2026

View reviewed changes

aboucaud marked this pull request as draft October 22, 2026 16:55

aboucaud mentioned this pull request Oct 22, 2026

Add the agentic eval on three harnesses, through lightcone-cli's Harbor workflow LightconeResearch/agent-skills#56

Open

aboucaud marked this pull request as ready for review October 22, 2026 17:27

aboucaud requested a review from EiffL October 22, 2026 17:28

aboucaud mentioned this pull request Oct 22, 2026

Eval: read the task's own facts and rubric #257

Closed

Eval: read lightcone-evals without a token, now that it is public

8b7e880

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

This was referenced Oct 23, 2026

snae-build: say the run is unattended LightconeResearch/lightcone-evals#4

Merged

Host the eval workflow beside the tasks it runs LightconeResearch/lightcone-evals#5

Merged

aboucaud changed the title ~~Run the agentic eval through Harbor, with a step gate and trace analysis~~ Run the agentic eval through the Harbor workflow hosted in lightcone-evals Oct 23, 2026

Eval: no step gate, so no actions: read

4f48c24

The hosted workflow no longer reads main's history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

aboucaud mentioned this pull request Oct 25, 2026

[CI maint]: Node 24 action versions, no uv cache LightconeResearch/lightcone-evals#7

Open

Merge main into eval-harbor

1342372

Keep main's ui/ structure line and drop the evals/ line, which this branch removes along with the old eval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

Navigation Menu

Repository navigation

Run the agentic eval through the Harbor workflow hosted in lightcone-evals - #255

Run the agentic eval through the Harbor workflow hosted in lightcone-evals#255
aboucaud wants to merge 11 commits into
mainfrom
eval-harbor

aboucaud commented Oct 22, 2026 •

edited

Loading

Uh oh!

aboucaud commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

aboucaud commented Oct 22, 2026

Uh oh!

aboucaud commented Oct 22, 2026

Uh oh!

aboucaud commented Oct 22, 2026

Uh oh!

aboucaud Oct 22, 2026

Uh oh!

EiffL commented Oct 22, 2026 •

edited

Loading

Uh oh!

aboucaud commented Oct 23, 2026

Uh oh!

aboucaud commented Oct 23, 2026

Uh oh!

aboucaud commented Oct 23, 2026

Uh oh!

aboucaud commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

aboucaud commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Acknowledgements

What

Decided

Verified

Uh oh!

aboucaud commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Eval

Uh oh!

aboucaud commented Oct 22, 2026

Uh oh!

aboucaud commented Oct 22, 2026

Which agent-skills version should the eval install?

What "published" means for agent-skills

The weakness of following main as it moves

Options

Proposal

Uh oh!

aboucaud commented Oct 22, 2026

Uh oh!

aboucaud Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

EiffL commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aboucaud commented Oct 23, 2026

Uh oh!

aboucaud commented Oct 23, 2026

Uh oh!

aboucaud commented Oct 23, 2026

Uh oh!

aboucaud commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aboucaud commented Oct 22, 2026 •

edited

Loading

github-actions Bot commented Oct 22, 2026 •

edited

Loading

The weakness of following `main` as it moves

EiffL commented Oct 22, 2026 •

edited

Loading