Skip to content

perf(sync): fetch documentation and skill files concurrently - #76

Merged
kieronlanning merged 1 commit into
mainfrom
chore/sync-parallel-fetch
Nov 16, 2026
Merged

kieronlanning merged 1 commit into
mainfrom
chore/sync-parallel-fetch

Conversation

@kieronlanning

Copy link
Copy Markdown
Contributor

Why

A live data sync is roughly 500 serial HTTPS round trips: one raw file per documentation page plus one commit-feed request per page (for lastModified), one tree listing per skills-declaring project, and one raw file per mirrored skill and asset. Its wall time was the sum of every request latency, which is why a single sync measured 196s of the deploy build job and is now the largest remaining cost in both pipelines.

What changed

Change Effect
mapWithConcurrency (new, src/src/lib/concurrency.ts) - bounded and order-preserving a channel pages are fetched eight at a time instead of one at a time
fetchRawFileContent exported from the docs aggregator the skill and asset mirrors need the bytes but never a page lastModified, so they stop issuing a commit-feed request per file and discarding the answer - about 160 requests per live sync
Unit tests for the helper order, limit, empty input, rejection

Ordering is preserved exactly - the helper writes each result at its input index - so page order, slug aliases and the sync log still follow repository tree order.

Evidence

A/B of DATA_MODE=live bun run data:sync, same machine, back to back:

wall time
before 210.6s
after 140.0s
-33%

The gain scales with request latency, so on a CI runner (where the same sync measured 196s) it should be larger than this.

Also verified: astro check 0 errors / 0 warnings, 72 unit tests pass, oxlint and oxfmt clean, and just validate green in 43s - which includes check-generated and test-dist against the mirror this code produced (SEO check passed: 365 indexable page(s), Link check passed).

Follow-ups (measured, not in this PR)

  • syncSkills still walks 108 skills and their files one at a time; fanning that out with the same helper is the next comparable win.
  • The docs aggregation and the skills mirror each list every repository tree (18 + 13 calls per sync); a per-process memo keyed repository@ref would remove the duplicates.

A live data sync was dominated by serial HTTPS round trips: one raw file per documentation page plus one commit-feed request per page (for `lastModified`), one tree listing per skills-declaring project, and one raw file per mirrored skill and asset - all issued one at a time. A full sync is roughly 350 requests for the documentation and 160 for the skills, so its wall time was the sum of every request latency.

Adds `mapWithConcurrency` (bounded and order-preserving) and uses it for a channel pages, so eight-way fetching hides per-request latency without stampeding GitHub raw and commit-feed endpoints. Exports `fetchRawFileContent` for callers that need the bytes but not a page lastModified, and switches the skill and asset mirrors to it, removing a second request per mirrored file whose result was discarded. Unit tests cover the helper ordering, limit, empty input and rejection.

Ordering is preserved exactly: page order, slug aliases and the sync log still follow the repository tree order.
@kieronlanning
kieronlanning enabled auto-merge (squash) November 16, 2026 12:19
@kieronlanning
kieronlanning merged commit 5fdac0b into main Nov 16, 2026
1 check passed
@kieronlanning
kieronlanning deleted the chore/sync-parallel-fetch branch November 16, 2026 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant