Repository navigation
Evals: isolate Codex from the runner's own setup; record the post-rename eval - #134
Conversation
…rename eval A Codex eval loaded the runner's ~/.codex config: it called the Computer Use MCP server on the host, ran hooks and tried to update an unrelated plugin marketplace, then took 15 minutes to exit. --ignore-user-config, --ignore-rules, --ephemeral and hooks off keep that out; the same run now passes in 192s instead of 1097s. Claude and Codex both pass all checks after the ROUTE.md rename. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 54 minutes. View limit details
 |
| Layer / File(s) | Summary |
|---|---|
Codex runner options and documentation evals/run.mjs, evals/README.md |
The Codex invocation adds options to disable hooks, ignore user configuration and rules, and run ephemerally. The README notes that login still uses ~/.codex and global instruction files remain active. |
Evaluation results docs/research/2026-11-03-eval-after-route-rename.md |
The report records tested versions and runs, the observed effects of user configuration, the isolated rerun results, remaining global-instruction exposure, and evaluation limitations. |
Estimated code review effort: 2 (Simple) | ~10 minutes
Change: Bug fix
Possibly related issues
YuqingNicole/variant-design-skill#18: The report records a real-project evaluation after theROUTE.mdrename, which provides partial evidence for artifact-specific validation.YuqingNicole/variant-design-skill#24: The report records real Codex and Claude runs that verify skill discovery and use, supporting host-environment validation.YuqingNicole/variant-design-skill#25: The report records a first-run prompt in a real Vite/React project and generated output after the route rename.
Possibly related PRs
- YuqingNicole/variant-design-skill#121: This PR records real Claude and Codex evaluations following the
ROUTE.mdrename. - YuqingNicole/variant-design-skill#65: This PR updates the same evaluation runner and README to change Codex command behavior.
Merge Risk: 
d9126
-
The isolation changes are mergeable with a bounded compatibility caveat: older Codex installations may fail evals. Documenting a compatible CLI version would make setup more reliable.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat… Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely identifies both main changes: isolating Codex from the runner's setup and recording the post-rename evaluation. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 1 files. (2 skipped: 2 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches
-
Generate docstrings
-
- Commit to this branch
- Create a new PR
Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
A rabbit checked the flags at dawn,
Then watched the Codex hooks switch off.
The tests ran on, the notes were clear,
Global instructions still stayed near.
I tucked the report beneath my ear.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
…cked with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why
I wanted to confirm that renaming
skills/*/SKILL.mdtoROUTE.md(#121) didn't break real runs. While checking, I found that Codex eval runs were not isolated from the machine running them.Results (pulse; details in
docs/research/2026-11-03-eval-after-route-rename.md)All three runs read
skills/variant-generate/ROUTE.md, so routing works after the rename.The isolation problem
With the old command, Codex loaded the runner's
~/.codex:cua_repl) on the host machine to check the preview environment.evals/run.mjsnow passes--ignore-user-config --ignore-rules --ephemeral -c features.hooks=false. I verified this first with a probe that runs no commands: Computer Use is no longer exposed, hooks no longer load, and login still works. Then I re-ran the full eval.Still not isolated: both CLIs read the runner's global instructions (
~/.codex/AGENTS.md,~/.claude/CLAUDE.md). I found no flag to skip them. The README now says so.Checks that ran
node --test evals/grade.test.mjs: passed.🤖 Generated with Claude Code
Summary by CodeRabbit