Skip to content

fix: scope upstream cooldowns and exempt internal forwards - #445

Merged
worstell merged 4 commits into
mainfrom
fix/cooldown-scope
Nov 10, 2026
Merged

worstell merged 4 commits into
mainfrom
fix/cooldown-scope

Conversation

@worstell

Copy link
Copy Markdown
Contributor

Requests that a strategy forwards to another cachew route (Hermit GitHub releases) skip cooldowns. The target route applies cooldowns to the real upstream. Before this change, one 429 on the loopback blocked every Hermit release request.

GitHub App installation tokens, the App JWT and the static GitHub token each get their own cooldown, because GitHub limits each of those credentials separately. All other requests keep the per-hostname cooldown. The scope is set where the credential is attached, not taken from a hash of the Authorization header, because a header hash splits one rate limit into many cooldowns when tokens rotate or are scoped per repository (OCI).

upstream-backoff-max (default 15m) caps the cooldown. Each cooldown start or extension writes a WARN log and increments cachew.upstream.cooldowns. Each local 429 increments cachew.upstream.cooldown_rejections.

Requests that a strategy forwards to another cachew route no longer
record or check cooldowns. The target route applies them to the real
upstream, so one 429 cannot pause every route behind the same address.

GitHub App installation tokens, App JWTs and the static GitHub token
get their own cooldown on each host, because GitHub limits each of
those credentials separately. All other requests keep the shared
per-hostname cooldown.

Cap cooldowns with upstream-backoff-max (default 15m). Log each
cooldown that starts or is extended, and count cooldowns and local
429 rejections.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1c2b1-2571-713a-816e-92bc6af0460d
Co-authored-by: Amp <amp@ampcode.com>
@worstell
worstell marked this pull request as ready for review November 10, 2026 02:00
@worstell
worstell requested a review from a team as a code owner November 10, 2026 02:00
@worstell
worstell requested review from joshfriend and removed request for a team November 10, 2026 02:00
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T03:11:26.185364Z 4e7d7e4 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59ff59592c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cmd/cachewd/main.go Resolved
Comment thread internal/strategy/git/git.go Resolved
… to the token's owner

Raise an upstream-backoff-max below upstream-backoff to upstream-backoff
instead of failing startup, so configs that set only a long
upstream-backoff keep working.

Scope git proxy cooldowns to the owner whose installation minted the
token, so orgs served by the fallback installation share its cooldown.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1c2b1-2571-713a-816e-92bc6af0460d
Co-authored-by: Amp <amp@ampcode.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e992188c70

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/githubapp/tokens.go Resolved
Comment thread internal/githubapp/transport.go Resolved
GitHub limits each App's JWT requests and each installation's token
separately, and one owner can have installations of several Apps. Scope
installation-token cooldowns by installation ID, which is unique across
Apps, and App JWT cooldowns by App ID, so a 429 for one App or
installation does not pause another.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1c2b1-2571-713a-816e-92bc6af0460d
Co-authored-by: Amp <amp@ampcode.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8ef4188cf9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/strategy/github_releases.go Resolved
Go keeps the request context but removes Authorization when it follows
a redirect to another host. Apply a cooldown scope only while the
request carries Authorization, so an anonymous hop to an asset CDN
shares that host's cooldown with every other caller.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1c2b1-2571-713a-816e-92bc6af0460d
Co-authored-by: Amp <amp@ampcode.com>
@worstell
worstell merged commit 82ad36b into main Nov 10, 2026
7 checks passed
@worstell
worstell deleted the fix/cooldown-scope branch November 10, 2026 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants