Repository navigation
perf(backend): batch tag date lookups instead of one request per tag - #535
Open
stefan-ernst wants to merge 2 commits into
Open
stefan-ernst wants to merge 2 commits into
stefan-ernst wants to merge 2 commits into
Conversation
ListTags resolved each tag's date with its own GetCommit call, and only then did the caller apply the milestone tag pattern. A repository with ~1,200 tags issued hundreds of commit requests every sync, exhausted the GitHub rate limit, and stretched each sync to minutes (#534). GitHub now reads tag names, SHAs, and dates from one paginated GraphQL query, narrowing the ref query to the pattern's literal prefix. GitLab and Gitea already carry dates in their tag listings, so they filter by name without any extra request. The unused `since` cutoff and the 500-tag truncation cap are gone, and the pattern now reaches the provider so non-matching tags are never resolved.
…edger Follow-up to the batched tag lookup. The tag path ignored last_synced_at and re-listed the whole tag history every sync, and scm_processed_refs was read once per tag. Pass the checkpoint (minus a 7-day lookback) as a refetch cutoff. GitHub orders refs by TAG_COMMIT_DATE desc and stops paginating at the first tag past the cutoff, returning it as the predecessor anchor so prev_tag still resolves. GitLab and Gitea apply the same cutoff after listing. Load the processed-ref set once per sync instead of one query per tag.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the tag-scanning cost reported in #534.
ListTagsresolved each tag's date with its ownGetCommitcall, and themilestone tag pattern was only applied after every tag had been resolved. A
repository with ~1,200 tags issued hundreds of commit requests per sync,
exhausted the GitHub rate limit, and stretched each sync to minutes.
Changes
Batch the lookup
(
refs(refPrefix: "refs/tags/")) instead of oneGET /commits/{sha}per tag.The ref query is narrowed to the pattern's literal prefix, and the exact glob
match stays authoritative client-side.
by name and no longer need a fallback commit fetch (Gitea).
Date stamp (refetch cutoff)
last_synced_atand re-listed the whole tag historyevery sync.
syncTagsAndReleasesnow passes the checkpoint minus a 7-daylookback as a cutoff, mirroring
iteratePullRequests/syncCommits.TAG_COMMIT_DATEdescending and stops paginating at thefirst tag past the cutoff. That tag is returned as the predecessor anchor so
ref.prev_namestill resolves for the oldest in-window tag. GitLab and Giteaapply the same cutoff after listing (
applyTagCutoff).last_synced_atnull) keeps its cutoff at zero and baselinesthe whole set, unchanged.
Refetch guard
scm_processed_refswas read once per tag.loadProcessedRefsloads the setonce per sync and checks membership in memory.
longer silently dropped.
Verification
./overlay.sh ../core-scm-tags -- -tags=test -count=1 ./internal/scm/...— pass./overlay.sh ../core-scm-tags -- -tags=test -count=1 ./internal/server/... ./internal/services/...— passscripts/run-golangci-lint.sh run— 0 issuesTests: Windshiftapp/core-tests-internal#59
Notes
refs(query:)prefix andorderBy: TAG_COMMIT_DATEwereconfirmed against the live GitHub GraphQL schema. The client-side glob match
stays authoritative, so a prefix superset is safe.
that was never processed (an old commit tagged later) would be dropped. The
7-day lookback bounds that; the ledger covers everything inside the window.
GitHub stops paginating early. They were not the reported bottleneck.