Skip to content

perf(sync): mirror skills concurrently and keep the offline modes offline - #77

Merged
kieronlanning merged 1 commit into
mainfrom
chore/sync-followups
Nov 16, 2026
Merged

kieronlanning merged 1 commit into
mainfrom
chore/sync-followups

Conversation

@kieronlanning

Copy link
Copy Markdown
Contributor

Why

Four follow-ups to #76, all the same shape: work that was serial, repeated, or unnecessary.

What changed

1. The skills mirror is fetched concurrently. syncSkills fetched one file at a time, per skill - 108 skills, ~160 requests, each waiting for the last. It now plans first (which ref, cached or not, which files), fetches the flattened list eight at a time, then writes, marks and reports in plan order, so the mirrored tree and the log are identical to the serial result.

2. cache and fixture are offline again. They promise a sync that touches no network (data-sync.ts), but the skills mirror listed every repository tree before requiring the cache - so the documented offline loop (DATA_MODE=fixture just build) made ~13 REST calls. Those modes now take the skill set from the cache (cachedSkills): the .ref marker beside each cached skill names the ref it was fetched from, and a declared skill that was never fetched still fails loudly instead of vanishing.

3. auto stops looking up what it cannot need. A project whose declared skills are all cached and that asks for no discovery (skills: list rather than skills: true) no longer lists its tree: nothing to fetch, nothing to discover.

4. One tree listing per repository, not two. listRepositoryFiles is memoized per repository@ref for the process, because a sync lists most trees twice - once for the documentation channels, once for the skills mirror. A failed listing is not cached, so a transient error is retried rather than remembered.

The 3rd item on my follow-up list is a documentation change, not a code change. The PR gate and the main deploy run the same chain deliberately; skills/site-validation-loop now says so, and says that removing the second run is a decision about what the deployment no longer has to prove. For the record, release.yml is already path-filtered to package.json, so main does not run a second full pipeline on a content push.

Evidence

A/B of DATA_MODE=live bun run data:sync, same machine, same work, same 108 skills:

version wall time
original 210.6s
+ parallel documentation fetch (#76) 140.0s
+ this PR 74.6s

-65% overall, -47% on top of #76.

Verified: 452 unit tests pass (2 new for the cached skill set), astro check 0 errors / 0 warnings, oxlint and oxfmt clean, and just validate green in 44s - which includes check-generated and test-dist against the skills mirror and documentation this code produced.

Not done, and why

  • The remaining duplication is the two gates themselves (PR + main). Sharing one run between them would mean publishing on the strength of a PR-time validation, on a different runner, against different live data. That is a policy decision, not a refactor, so it is written down rather than assumed.
  • skills: true means "publish every skill under skills/", so in auto/live the repository tree is the source of truth for which skills exist and must stay listed. Only the cache-only modes can answer without it.

@kieronlanning
kieronlanning enabled auto-merge (squash) November 16, 2026 13:24
…line

Follow-up to the concurrent documentation fetch:

syncSkills fetched one file at a time, per skill, for 108 skills (~160 requests). It now plans the work first - which ref, cached or not, which files - fetches the flattened list eight at a time, and only then writes, marks and reports, in plan order, so the mirror and the log are identical to the serial result.

cache and fixture promised a sync that touches no network but listed every repository tree anyway before requiring the skills cache. They now take the skill set from the cache (cachedSkills): the per-skill .ref markers say what was fetched and at which ref, and a per-project marker records the ref the project was last synced for. That marker is what tells an empty skill set apart from one that was never fetched, so an offline sync neither makes a request nor goes quiet when the cache is cold.

auto no longer lists a tree for a project whose declared skills are all cached and that asks for no discovery - there is nothing to look up.

listRepositoryFiles is memoized per repository and ref for the process, so a sync stops listing the same tree twice (once for the documentation channels, once for the skills mirror).

Measured on one machine: a live sync goes from 140.0s to 74.6s (210.6s before the first concurrent-fetch change), an auto sync with a warm cache to 7.1s, and a cache-mode sync to 1s. Unit tests cover the cached skill set (ref filtering, stray files, an absent cache directory) and the project ref marker.
@kieronlanning
kieronlanning merged commit 474c0c4 into main Nov 16, 2026
1 check passed
@kieronlanning
kieronlanning deleted the chore/sync-followups branch November 16, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant