Skip to content

chore(ci): harden the release supply chain - #128

Merged
dduugg merged 8 commits into
mainfrom
chore/release-supply-chain
Nov 13, 2026
Merged

dduugg merged 8 commits into
mainfrom
chore/release-supply-chain

Conversation

@perryqh

@perryqh perryqh commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Updated to address the review. The branch merges main, and each fix from the review is its own commit. Perry's original description is kept at the end. Its notes on SHA pinning, Dependabot and the if: fix describe work main has since done in #136, #137 and #139.

Summary

Hardens how codeowners releases are built and published, and documents how to use them safely through DotSlash.

What this adds to main

  • Draft, then publish. The release job creates a draft. The upload jobs package, attest and upload each archive, generate-dotslash-files writes the DotSlash file, and publish-release publishes last. So a release is complete before anyone can see it, which immutable releases require.
  • Build provenance attestations for each release archive, made after packaging so they cover what users download.
  • macOS: an ad-hoc re-sign after lipo, checked with codesign --verify.
  • Re-runs: an existing draft is reused, uploads use --clobber, and a DotSlash file left by an earlier attempt is deleted before it's regenerated.
  • DotSlash config: exact asset names, exclude-http-provider, since a draft's asset URLs are untagged-... paths that would be written into the file dead, and SHA-256 digests, so the digest in a committed DotSlash file is the one gh attestation verify checks.
  • Reproducibility: cross is pinned to 0.2.5, and release builds use --locked.
  • Concurrency: one group per commit on main, so a later push can't cancel a queued version bump. Elsewhere a newer push cancels the older run.
  • README: commit the DotSlash file and upgrade through a reviewed diff, install the gh CLI for DotSlash files after v0.6.0, verify attestations, and understand macOS signing and quarantine.

Commits since the review

  • 79abe63 merges main. Main's pins (actions/checkout v7.0.1 with persist-credentials: false, dotslash-publish-release v2), its license notices in every archive, and its dependabot.yml and audit.yml are kept. This PR no longer changes those two files.
  • 85c0359 deletes a DotSlash file left by an earlier attempt, since the action uploads without --clobber and the archives it describes may have been replaced.
  • 389c090 gives each commit on main its own concurrency group.
  • 70bc111 fails the draft step on any gh release view error other than "release not found", rather than creating a duplicate draft.
  • 90c78f0 updates the README: the gh requirement, attestation history through v0.6.0, and the signing wording.
  • 553e41f records SHA-256 digests in the DotSlash file.
  • a92fb98 builds release binaries with --locked.

Worth knowing

  • The release job runs on ubuntu and no longer builds before creating the draft, so a macOS build failure leaves a draft behind. Re-running reuses it.
  • DotSlash files for releases after v0.6.0 have only the github-release provider, so consumers need the gh CLI on PATH.

Testing

  • zizmor (offline) reports nothing, as on main. actionlint reports only the 4 existing SC2086 infos in version detection.
  • dotslash-config.json is valid JSON, and cargo build --release --locked succeeds.
  • The draft step and the DotSlash cleanup step were run against a stub gh in each case: a draft, a published release, no release and an API error for the first, and the asset present, absent and an API error for the second.
  • The release path only runs on main, so the workflow itself is untested beyond that. Perry's runs in a throwaway repo with immutable releases, described below, cover the draft-then-publish flow and predate these commits.

Not in this PR

  • Attesting in a separate job, so that the build doesn't run with id-token: write.
  • Developer ID signing and notarization for macOS.

Requires a manual step

Enable Settings → General → Immutable releases after merge.

Original description

Why

We distribute codeowners via DotSlash. Reading verify_artifact() settled what actually protects the binaries: DotSlash checks the size and BLAKE3/SHA-256 digest recorded in the DotSlash file, and nothing else. There is no signature verification anywhere in the tool.

So for the DotSlash path the digest is the mechanism, and it is only as trustworthy as the file holding it. That reframes the work around how that file is generated and consumed.

The main fix is documentation

README.md told users to download the DotSlash file from a release and run it. If you re-fetch the DotSlash file next to the binary every time, the digest verifies nothing — you're trusting whatever the release currently claims, and an attacker who can modify the release swaps both.

DotSlash's own motivation doc is explicit: "the provenance of a DotSlash file is paramount" and DotSlash files "are meant to live in source control." The README now says to commit the file and upgrade via a reviewed diff, and explains why.

Pipeline changes

  • Draft → publish. The release is created with --draft and published only after every asset lands. Prerequisite for immutable releases: once published, gh release upload fails with Cannot upload assets to an immutable release (attest-build-provenance#734).
  • Build provenance via actions/attest-build-provenance, attested after packaging so the covered digest matches what users download.
  • exclude-http-provider on the DotSlash config. Generating against a draft bakes a permanently dead provider URL into the published file. A draft has no git tag, so GitHub serves assets under an untagged-<hash> path and the generator records that verbatim. Publishing rewrites the real URLs, but the DotSlash file is already uploaded — and under immutable releases it cannot be replaced. Found by running the pipeline, not by reading it (see Testing).
  • DotSlash assets matched by exact name instead of regex. ^codeowners-mac is prefix-only and unanchored, and the generator takes the first match over dict order. The generated DotSlash file is itself named codeowners, so on a re-run a prefix pattern could select it as the binary.
  • All third-party actions pinned to commit SHAs, plus .github/dependabot.yml so the pins don't go stale. Only codeql.yml stays on @main, which already carries a documented zizmor: ignore. cross pinned to 0.2.5.
  • Ad-hoc re-sign after lipo, per Apple DTS guidance, with codesign --verify so a regression fails in CI rather than for a user.
  • Idempotent re-runs. Reuse an existing draft rather than creating a duplicate (a draft creates no tag, so GitHub does not enforce tag uniqueness), --clobber on uploads, and a concurrency group that queues on main rather than cancelling mid-release.
  • Fixed if: success() && ${{ ... }} on the DotSlash job — mixing a bare expression with an interpolation doesn't evaluate as written.

Testing

Local: clippy, cargo fmt --check, and the full test suite pass (this PR touches no Rust).

The release path doesn't run outside main, so I ran the whole pipeline twice in a throwaway public repo with Immutable releases enabled:

  • dotslash-publish-release does read assets from a draft release. This was the main open question.
  • All 8 jobs pass in order: check/test/lints → release (draft) → both uploads + attest → generate-dotslash-files → publish-release.
  • The published release reports draft: false, immutable: true with all four assets, confirming the ordering is what makes immutable releases workable.
  • gh attestation verify passes against the published macOS tarball.
  • The generated DotSlash file executes: dotslash codeowners --version → codeowners 0.3.6. That covers the consumer path end to end — provider resolution, size and BLAKE3 enforcement, unpack, run. It also confirms the post-lipo re-sign is valid, since unsigned code is SIGKILLed on Apple silicon rather than failing quietly.
  • The second run had a prior release present, so --generate-notes and the resolve-draft-by-pending-tag path were exercised with history rather than on an empty repo.

The first run is what surfaced the untagged- provider bug: it shipped a DotSlash file whose primary provider 404'd on all four platforms. After exclude-http-provider, the regenerated file carries exactly one github-release provider per platform.

Still unverified: the re-run-after-partial-failure path. The idempotency logic is written for it, but a green run never takes that branch.

Known gap: Developer ID signing

macOS binaries are ad-hoc signed only. Ad-hoc signing satisfies Apple silicon's requirement that all code be signed, but it asserts nothing about who built the binary and does not satisfy Gatekeeper — so a browser download (which sets com.apple.quarantine, unlike DotSlash or curl) is blocked. The README now documents this and the xattr -d workaround.

Real Developer ID signing plus notarization is the fix, and Gusto/gusto-cli already does it from a Linux runner via rcodesign. Porting that here needs a decision about where the signing credential lives, since this is a public repo outside the Gusto org. Tracked separately from this PR.

Requires a manual step

Enable Settings → General → Immutable releases after merge. The draft→publish restructure exists to make that safe; before this change, enabling it would have broken the next version bump.

@perryqh

perryqh commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Applied the review findings in 81a4041:

  • Idempotent draft creation. Reuse an existing draft rather than creating a duplicate. A draft creates no git tag, so GitHub does not enforce tag uniqueness for drafts, and gh's lookup returns the first draft release that has tagName as its pending tag — reachable via "Re-run all jobs" after a failed upload.
  • Concurrency group. Queue on main (cancelling mid-release would strand a partial draft), cancel elsewhere via cancel-in-progress: ${{ github.ref != 'refs/heads/main' }} so PR feedback stays fast.
  • --clobber on all three asset uploads.
  • Pinned cargo install cross --version 0.2.5 — the largest unpinned input left in the release path, and what an unpinned install resolves to today, so no behavior change. Deliberately not --locked: cross 0.2.5's lockfile predates current rustc and this build cannot be exercised outside main.
  • README now states that releases up to and including v0.3.4 predate attestation, so gh attestation verify reporting no match for them is expected rather than a tampering signal. Avoided naming a future version, which would go stale if the next bump is a minor.
  • Dropped the redundant codesign -dv.

I verified the idempotency shell logic against all three states (no release / existing draft / existing published release), including that set -e does not abort on the failed lookup inside the condition.

Not applied, deliberately:

  • actionlint in CI — I could not download it to verify it passes against the current workflows, and adding a gate I have not run is how you get a red main. Worth a follow-up.
  • The grep "\\+version" fragility in version detection — a [dependencies.foo] section with a version key would extract the wrong string and cut a bogus release. Not currently triggerable (Cargo.toml has no such sections), and it is the riskiest line in the file to touch in a PR whose release path CI cannot exercise. Follow-up issue.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the release supply chain for codeowners distribution via DotSlash by making the DotSlash file’s provenance reviewable and by strengthening the CI release workflow (draft-first publishing, provenance attestations, and pinned actions).

Changes:

  • Updated README guidance to commit the DotSlash file in-repo and explain why that’s necessary for digest verification to be meaningful.
  • Restructured the release workflow to create draft releases first, upload/attest packaged assets, generate DotSlash files, then publish the release.
  • Pinned third-party GitHub Actions to commit SHAs and added Dependabot configuration to keep those pins updated.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
README.md Updates DotSlash usage and adds guidance for independently verifying release provenance.
.github/workflows/dotslash-config.json Switches DotSlash generator selection from regex matching to exact asset names.
.github/workflows/ci.yml Implements draft→upload/attest→dotslash→publish flow; adds concurrency control; pins actions to SHAs.
.github/workflows/audit.yml Pins actions to SHAs for the security audit workflow.
.github/dependabot.yml Adds Dependabot updates for GitHub Actions SHA pins.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread README.md Resolved
@perryqh
perryqh marked this pull request as ready for review August 26, 2026 22:44
@perryqh
perryqh requested a review from a team as a code owner August 26, 2026 22:44
@perryqh
perryqh marked this pull request as draft August 26, 2026 23:36
Binaries are distributed via DotSlash, which verifies only the `size` and
BLAKE3 `digest` recorded in the DotSlash file -- there is no signature check
anywhere in the tool. The digest is the whole mechanism, and it is only as
trustworthy as the file holding it. This reworks how that file is produced
and consumed, and closes the gaps around it.

README:

- Commit the DotSlash file and upgrade via a reviewed diff. Re-fetching it
  next to the binary means trusting whatever the release currently claims,
  so the digest verifies nothing. This is DotSlash's own guidance.
- Document `gh attestation verify`, and note that ad-hoc signing does not
  satisfy Gatekeeper for browser downloads.

Pipeline:

- Create the release as a draft, publish only once every asset has landed.
  Required for immutable releases, which reject uploads to a published
  release.
- Attest build provenance after packaging, so the digest the attestation
  covers matches what users download.
- Pin third-party actions to commit SHAs; add dependabot to keep the pins
  fresh. Pin `cross` to 0.2.5.
- Match DotSlash assets by exact name. An unanchored prefix regex could
  select the generated DotSlash file itself as a binary on a re-run.
- Set `exclude-http-provider`. Generating against a draft bakes an
  `untagged-<hash>` asset URL into the published file; it 404s once the real
  tag exists and cannot be corrected under immutable releases.
- Re-sign after `lipo` so the universal binary carries one coherent ad-hoc
  signature, and verify rather than assume.
- Reuse an existing draft instead of creating a duplicate, `--clobber` on
  uploads, and a concurrency group that queues on main.

Known gap: macOS Developer ID signing and notarization. Ad-hoc signing
asserts nothing about who built the binary; provenance currently comes from
the attestations. Tracked separately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@perryqh
perryqh force-pushed the chore/release-supply-chain branch from 7d3c19b to 1f17a23 Compare August 26, 2026 23:44
@dduugg dduugg mentioned this pull request Sep 29, 2026
3 of 4 tasks

@dduugg dduugg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Reading DotSlash's verify_artifact() to pin down what actually protects the binaries, then running the whole pipeline twice in a throwaway repo with immutable releases on, is far more than most CI changes get, and it's what caught the untagged- provider bug. The README point that a DotSlash file only protects anything if it's committed and reviewed is the most valuable change here.

Most of this still stands, but the branch was cut before #136, #137, #139 and #142, so it needs a rebase, and two of the new pieces have failure modes worth fixing first. Details are inline. In short:

  • Rebase. Main already has SHA pins (at newer versions), persist-credentials: false, zizmor, a broader dependabot.yml, and license notices in the release archives. Taking this branch's side of the conflicts would downgrade or drop each of those. Starting from main's ci.yml and reapplying the new pieces (draft-then-publish, attestation, draft reuse, --clobber, concurrency, the cross pin, the re-sign, the DotSlash config) is probably simpler than resolving the conflicts. audit.yml and dependabot.yml can be dropped from the PR.
  • The re-run path that the description lists as unverified can publish an immutable release with a wrong DotSlash digest (on ci.yml, the dotslash step).
  • The concurrency group can cancel a pending version-bump run, so that version never gets released.
  • exclude-http-provider makes the gh CLI a requirement for every DotSlash consumer, which the README should say.

A few things with no line to attach to:

  • The release job moves from macos-latest to ubuntu-latest and drops its Toolchain info and Build steps. That seems fine, but it isn't in the description, and a broken macOS build now fails only after the draft exists, leaving it behind.
  • After the merge, main's "Release archives after v0.5.0 also hold a licenses directory" paragraph lands under "Verifying a release independently". It belongs back in the DotSlash section.
  • README line 96 says the macOS binary is "ad-hoc (linker) signed", but after codesign --force --sign - the signature comes from codesign.
  • Optional: setting "hash": "sha256" per platform in dotslash-config.json would make the digest in the committed DotSlash file the same one gh attestation verify checks, which connects the two halves of the README's provenance story. As it is, the attestation covers the tarballs and the DotSlash file uses BLAKE3, so a reader can't easily tie one to the other.
  • While here: --locked on the release cargo build and cross build lines would be a cheap win for a supply-chain PR.

Comment thread .github/workflows/ci.yml Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread .github/workflows/dotslash-config.json Resolved
Comment thread README.md Resolved
Main picked up much of this branch's hardening on its own (#136 pinned
the actions with persist-credentials: false and added zizmor, #137
moved dotslash-publish-release to v2, #139 added Dependabot for cargo
and GitHub Actions, #142 added license notices to the release
archives), so the conflicts are resolved by taking main's version and
reapplying only what is still new here: the draft release that is
published last, build provenance attestations, reuse of an existing
draft, --clobber on uploads, the concurrency group, the cross 0.2.5
pin, the ad-hoc re-sign after lipo, exact asset names and
exclude-http-provider in the DotSlash config, and the README's
provenance guidance.

dependabot.yml and audit.yml are main's, so they drop out of this
branch's diff. Every release archive still carries the licenses
directory, and attestation still runs after packaging, so it covers the
archive as uploaded. The README's note about the licenses directory
stays in the DotSlash section, where main has it.
dotslash-publish-release uploads the DotSlash file without --clobber.
After "Re-run all jobs" following a partial failure, the upload jobs
rebuild the archives and replace them, and their bytes can change since
tar and gzip record mtimes, but the action's upload of the new DotSlash
file then fails because the old one is still there. Under v1 that
failure was swallowed and publish-release went on to publish an
immutable release whose DotSlash file had the previous run's sizes and
digests. v2 checks the upload, so the job now fails instead, which is
safe but can't be re-run.

Delete an existing codeowners asset before the action runs. The asset
list is read first rather than matching delete-asset's not-found
message, so any other error from gh still fails the step.
With cancel-in-progress: false, a concurrency group holds one running
and at most one pending run, and a newer run cancels the pending one.
With one group for all of main, a version-bump commit waiting behind
an in-flight run was cancelled when another commit landed, and since
version detection only looks at the commit's own diff, that version
was never released.

On main the group is now the commit SHA, so every push runs. A re-run
of the same commit shares its group, and reuses the draft if one
exists. Other refs still cancel an in-flight run when a newer push
arrives.
The draft lookup sent gh's stderr to /dev/null and created a release
whenever the result wasn't "true", so a transient API error read as
"no draft" and created the duplicate draft the step exists to prevent.

gh prints exactly "release not found" when no release or draft has the
tag, so create the draft only then. Any other gh failure now fails the
step with gh's message, and a tag that is already published fails with
a clear error instead of whatever gh release create reports.
exclude-http-provider leaves github-release as the DotSlash file's only
provider, and DotSlash fetches through it by running gh release
download, so using a DotSlash file from a release after v0.6.0 needs
the GitHub CLI on PATH. Say so in the install steps.

v0.4.0, v0.5.0 and v0.6.0 shipped after the attestation note was
written, also without attestation, so it now names v0.6.0. And since
the merged binary is re-signed with codesign, its ad-hoc signature no
longer comes from the linker.
The attestation covers each release archive's SHA-256, while the
DotSlash file recorded BLAKE3 digests, so there was no easy way to
check that a committed DotSlash file pins the bytes that were attested.
dotslash-publish-release v2 and DotSlash both accept "hash": "sha256"
per platform, and with it the DotSlash digest and the attested digest
are the same value.

The README now says the DotSlash digest is of the release archive,
which is what DotSlash checks, rather than of the binary, and that for
releases after v0.6.0 it matches what gh attestation verify checks.
Without --locked, cargo build and cross build may update Cargo.lock
when it doesn't satisfy Cargo.toml, and the release binaries would then
be built from dependency versions that no commit records. With it, the
build fails instead, so the binaries are built from the lockfile that
was reviewed.

cargo install cross stays unlocked, as the comment on that step
explains.
@dduugg
dduugg marked this pull request as ready for review November 13, 2026 00:58
@dduugg
dduugg merged commit e2d18aa into main Nov 13, 2026
16 of 17 checks passed
@dduugg
dduugg deleted the chore/release-supply-chain branch November 13, 2026 01:06
@github-project-automation github-project-automation Bot moved this from Triage to Done in Modularity Nov 13, 2026
@dduugg dduugg mentioned this pull request Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants