Skip to content

perf(backend): batch tag date lookups instead of one request per tag - #535

Open
stefan-ernst wants to merge 2 commits into
mainfrom
fix/scm-batch-tag-lookups
Open

stefan-ernst wants to merge 2 commits into
mainfrom
fix/scm-batch-tag-lookups

Conversation

@stefan-ernst

@stefan-ernst stefan-ernst commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What

Fixes the tag-scanning cost reported in #534.

ListTags resolved each tag's date with its own GetCommit call, and the
milestone tag pattern was only applied after every tag had been resolved. A
repository with ~1,200 tags issued hundreds of commit requests per sync,
exhausted the GitHub rate limit, and stretched each sync to minutes.

Changes

Batch the lookup

  • GitHub: read tag names, SHAs, and dates from one paginated GraphQL query
    (refs(refPrefix: "refs/tags/")) instead of one GET /commits/{sha} per tag.
    The ref query is narrowed to the pattern's literal prefix, and the exact glob
    match stays authoritative client-side.
  • GitLab / Gitea: their tag listings already carry the date, so they filter
    by name and no longer need a fallback commit fetch (Gitea).

Date stamp (refetch cutoff)

  • The tag path ignored last_synced_at and re-listed the whole tag history
    every sync. syncTagsAndReleases now passes the checkpoint minus a 7-day
    lookback as a cutoff, mirroring iteratePullRequests / syncCommits.
  • GitHub orders refs by TAG_COMMIT_DATE descending and stops paginating at the
    first tag past the cutoff. That tag is returned as the predecessor anchor so
    ref.prev_name still resolves for the oldest in-window tag. GitLab and Gitea
    apply the same cutoff after listing (applyTagCutoff).
  • The first sync (last_synced_at null) keeps its cutoff at zero and baselines
    the whole set, unchanged.

Refetch guard

  • scm_processed_refs was read once per tag. loadProcessedRefs loads the set
    once per sync and checks membership in memory.
  • Removed the 500-tag truncation cap so repositories with more tags are no
    longer silently dropped.

Verification

  • ./overlay.sh ../core-scm-tags -- -tags=test -count=1 ./internal/scm/... — pass
  • ./overlay.sh ../core-scm-tags -- -tags=test -count=1 ./internal/server/... ./internal/services/... — pass
  • scripts/run-golangci-lint.sh run — 0 issues
  • Layering and handler-DB guards — pass

Tests: Windshiftapp/core-tests-internal#59

Notes

  • The GraphQL refs(query:) prefix and orderBy: TAG_COMMIT_DATE were
    confirmed against the live GitHub GraphQL schema. The client-side glob match
    stays authoritative, so a prefix superset is safe.
  • The cutoff is a hard skip: a tag whose commit date predates the lookback but
    that was never processed (an old commit tagged later) would be dropped. The
    7-day lookback bounds that; the ledger covers everything inside the window.
  • GitLab and Gitea still list all tags and apply the cutoff client-side; only
    GitHub stops paginating early. They were not the reported bottleneck.

ListTags resolved each tag's date with its own GetCommit call, and only
then did the caller apply the milestone tag pattern. A repository with
~1,200 tags issued hundreds of commit requests every sync, exhausted the
GitHub rate limit, and stretched each sync to minutes (#534).

GitHub now reads tag names, SHAs, and dates from one paginated GraphQL
query, narrowing the ref query to the pattern's literal prefix. GitLab
and Gitea already carry dates in their tag listings, so they filter by
name without any extra request. The unused `since` cutoff and the
500-tag truncation cap are gone, and the pattern now reaches the
provider so non-matching tags are never resolved.
@stefan-ernst stefan-ernst changed the title perf(scm): batch tag date lookups instead of one request per tag perf(backend): batch tag date lookups instead of one request per tag Nov 18, 2026
…edger

Follow-up to the batched tag lookup. The tag path ignored last_synced_at
and re-listed the whole tag history every sync, and scm_processed_refs was
read once per tag.

Pass the checkpoint (minus a 7-day lookback) as a refetch cutoff. GitHub
orders refs by TAG_COMMIT_DATE desc and stops paginating at the first tag
past the cutoff, returning it as the predecessor anchor so prev_tag still
resolves. GitLab and Gitea apply the same cutoff after listing. Load the
processed-ref set once per sync instead of one query per tag.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant