Repository navigation
perf(sync): fetch documentation and skill files concurrently - #76
Merged
Merged
Conversation
A live data sync was dominated by serial HTTPS round trips: one raw file per documentation page plus one commit-feed request per page (for `lastModified`), one tree listing per skills-declaring project, and one raw file per mirrored skill and asset - all issued one at a time. A full sync is roughly 350 requests for the documentation and 160 for the skills, so its wall time was the sum of every request latency. Adds `mapWithConcurrency` (bounded and order-preserving) and uses it for a channel pages, so eight-way fetching hides per-request latency without stampeding GitHub raw and commit-feed endpoints. Exports `fetchRawFileContent` for callers that need the bytes but not a page lastModified, and switches the skill and asset mirrors to it, removing a second request per mirrored file whose result was discarded. Unit tests cover the helper ordering, limit, empty input and rejection. Ordering is preserved exactly: page order, slug aliases and the sync log still follow the repository tree order.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A live data sync is roughly 500 serial HTTPS round trips: one raw file per documentation page plus one commit-feed request per page (for
lastModified), one tree listing per skills-declaring project, and one raw file per mirrored skill and asset. Its wall time was the sum of every request latency, which is why a single sync measured 196s of the deploy build job and is now the largest remaining cost in both pipelines.What changed
mapWithConcurrency(new,src/src/lib/concurrency.ts) - bounded and order-preservingfetchRawFileContentexported from the docs aggregatorlastModified, so they stop issuing a commit-feed request per file and discarding the answer - about 160 requests per live syncOrdering is preserved exactly - the helper writes each result at its input index - so page order, slug aliases and the sync log still follow repository tree order.
Evidence
A/B of
DATA_MODE=live bun run data:sync, same machine, back to back:The gain scales with request latency, so on a CI runner (where the same sync measured 196s) it should be larger than this.
Also verified:
astro check0 errors / 0 warnings, 72 unit tests pass, oxlint and oxfmt clean, andjust validategreen in 43s - which includescheck-generatedandtest-distagainst the mirror this code produced (SEO check passed: 365 indexable page(s),Link check passed).Follow-ups (measured, not in this PR)
syncSkillsstill walks 108 skills and their files one at a time; fanning that out with the same helper is the next comparable win.repository@refwould remove the duplicates.