ci: harden the release pipeline — isolate EKSCTLBOT_TOKEN, sign releases, publish SBOMs - #8863
Open
sapphirew wants to merge 2 commits into
Open
ci: harden the release pipeline — isolate EKSCTLBOT_TOKEN, sign releases, publish SBOMs#8863sapphirew wants to merge 2 commits into
sapphirew wants to merge 2 commits into
Conversation
The Draft Release Notes workflow ran as a single job that both handled
the release-drafter output and held EKSCTLBOT_TOKEN. That token is what
starts the publish pipeline (start-release.yaml triggers on a tag push
and passes it to publish-release.yaml, which runs GoReleaser and commits
the Homebrew formula), so it is the highest-value credential in the repo
and it does not need to be present while the drafter output is being
turned into a file.
Split the job in two:
render_release_draft contents: write, pull-requests: read
Runs release-drafter, renders the notes file and
the pull request body into RUNNER_TEMP, uploads
them as an artifact. No EKSCTLBOT_TOKEN, and no
checkout -- it does not need the working tree.
update_release_draft_pr contents: read
Checks out, copies the rendered notes into
docs/release_notes/, opens the pull request with
EKSCTLBOT_TOKEN. Consumes the drafter output only
as file content, via `body-path`, so it is never
interpolated into a run step.
Other changes this requires:
- `tag_name` is promoted to a job output. It comes from the
`tag-template: 'v$NEXT_MINOR_VERSION'` in .github/release-drafter.yml
and is only used in action inputs, not in a shell.
- The pull request body moves from an inline `body:` to `body-path:`,
rendered with printf. Verified byte-identical to what the previous YAML
template produced for the same tag and body.
- `add-paths: docs/release_notes` scopes the commit to the notes file.
- `pull-requests: write` drops to `read` on the drafter job.
.github/release-drafter.yml configures no `autolabeler:`, so nothing in
this workflow applies labels; the pull request labels are applied by
create-pull-request in the second job using its own token.
- `retention-days: 1` on the artifact, since it is only a job-to-job hand-off.
The `if: github.event.repository.fork == false` guard stays on the first
job; the second is gated transitively through `needs`.
`.github/.goreleaser.yml` had no `signs:` or `sboms:` block, so the only integrity signal on a released eksctl binary was `eksctl_checksums.txt`. That file establishes that an archive matches the checksum list, but not that the checksum list itself came from the eksctl release pipeline -- anything that can write to the release can rewrite both. Add keyless Sigstore signing over the checksum file. Because the checksum file already covers every archive, one signature covers the whole release. Keyless means the signing identity is the Actions OIDC token exchanged for a short-lived Fulcio certificate, so there is no long-lived signing key to store or rotate; verifiers pin the workflow identity instead: --certificate-oidc-issuer https://token.actions.githubusercontent.com --certificate-identity-regexp '^https://github.com/eksctl-io/eksctl/\.github/workflows/publish-release\.yaml@refs/tags/' This requires `id-token: write`, added to the `publish-release-candidate` job in start-release.yaml (that job is the only caller of publish-release.yaml, and reusable workflows inherit the caller's permissions). `contents: write` and `pull-requests: write` are unchanged. Also emit a CycloneDX SBOM per archive via syft, so the Go modules compiled into each binary are enumerable from the release itself. cosign and syft are installed as pinned steps before the GoReleaser steps; GoReleaser shells out to both by name. Both new blocks live in .goreleaser.yml, so the Homebrew release picks them up too via the existing `cat .goreleaser.yml .goreleaser.brew.yml` concatenation. Documented verification in userdocs/src/installation.md next to the existing checksum instructions.
This was referenced Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #8860. Two independent hardening changes to the release pipeline, kept as two separate commits so either can be dropped without losing the other:
dcf0a94release-drafter.yamlso the job rendering drafter output no longer holdsEKSCTLBOT_TOKENd06fe41Supersedes #8861 and #8862, which I've closed in favour of this single review.
1. Keep
EKSCTLBOT_TOKENout of the release-notes rendering jobThe Draft Release Notes workflow runs as one job that both handles the release-drafter output and holds
EKSCTLBOT_TOKEN. Those two things don't need to be in the same job.EKSCTLBOT_TOKENis the highest-reach credential in the repo:start-release.yamltriggers on a tag push and passes it topublish-release.yaml, which runs GoReleaser and commits the Homebrew formula toweaveworks/homebrew-tap. UnlikeGITHUB_TOKEN, a tag pushed with a PAT does start workflows. Meanwhile the same job does the string handling onsteps.draft.outputs.body, which is assembled from merged PR titles — so the least trustworthy input in the workflow sits in the job with the most reach.#8860 fixed the shell quoting. This separates the two concerns structurally, so the rendering step has nothing valuable within reach even if something about it regresses later.
render_release_draftupdate_release_draft_prcontents: write,pull-requests: readcontents: readEKSCTLBOT_TOKENpersist-credentials: falseRUNNER_TEMP, uploads as artifactbody-pathSupporting changes:
tag_namepromoted to a job output. It comes fromtag-template: 'v$NEXT_MINOR_VERSION'in.github/release-drafter.ymland is only used in action inputs (commit-message,title), never in a shell.body:tobody-path:, rendered withprintf.body-pathhas been available since create-pull-request v6; this repo is on v8.1.1.add-paths: docs/release_notesscopes the commit to the notes file.pull-requests: write→readon the drafter job..github/release-drafter.ymlconfigures noautolabeler:, so nothing in this workflow applies labels; the PR's own labels are applied by create-pull-request in the second job with its own token. Please sanity-check this one — it's the only permission I reduced based on what the config does rather than what the workflow obviously needs, and it's a one-line revert if I have it wrong.retention-days: 1on the artifact, since it's only a job-to-job hand-off.if: github.event.repository.fork == falsestays on the first job; the second is gated transitively vianeeds.2. Sign releases and publish SBOMs
.github/.goreleaser.ymlhas nosigns:orsboms:block, so the only integrity signal on a released binary iseksctl_checksums.txt. That file establishes an archive matches the checksum list, but not that the checksum list came from this pipeline — checksums and archives sit side by side in the same release, so whatever can write one can rewrite the other.Keyless Sigstore signing over the checksum file. One signature covers the release, because the checksum file already covers every archive. Keyless means the identity is the Actions OIDC token exchanged for a short-lived Fulcio certificate — no long-lived key to store or rotate. Verifiers pin the workflow identity instead:
This needs
id-token: write, added to thepublish-release-candidatejob instart-release.yaml— the only caller ofpublish-release.yaml(reusable workflows inherit the calling job's permissions).contents: writeandpull-requests: writeunchanged.sboms:uses GoReleaser's defaults, shelling out tosyftfor<archive>.sbom.jsonin CycloneDX JSON.cosignandsyftare installed as pinned steps before the GoReleaser steps. Both new blocks live in.goreleaser.yml, so the Homebrew release picks them up through the existingcat .goreleaser.yml .goreleaser.brew.ymlconcatenation. Verification documented inuserdocs/src/installation.md.Checklist
area/nodegroup) and target version (e.g.version/0.12.0)Note
I can't set labels on this repo. The
Enforce a valid PR categorycheck needs one ofkind/improvement/kind/feature/kind/bug/kind/docs/area/tech-debt/skip-release-notes—kind/improvementis the right one. Could a maintainer add it?Testing
PR body rendering — the part most likely to regress silently, since it moved from a YAML template to
printf. Rendered both ways for the same tag and body and diffed, reading the old template straight fromgit show 8e1d569:.github/workflows/release-drafter.yaml:Also confirmed the notes file lands at
docs/release_notes/0.220.0.mdfor tagv0.220.0, and that a body containing- evil"; whoami; echo " (#124)is written verbatim with nothing executed.Job graph and permissions:
GoReleaser config —
goreleaserwouldn't install in my environment (a transitive dependency fetch is blocked), so I validated against GoReleaser's published JSON schema instead, the same schemagoreleaser checkuses:Both the standalone config and the
cat-concatenated Homebrew config validate;artifacts: archiveis permitted forsbomsandartifacts: checksumforsignsper the schema enums.What I could not verify locally, and would ask a maintainer to confirm on the first tag: that the OIDC exchange succeeds end to end — i.e.
id-token: writepropagates through the reusable-workflow call and cosign gets its certificate. If it doesn't, the symptom is a GoReleaser failure in the signing stage rather than a silently unsigned release. Worth exercising on an-rc.tag first; the RC path uses the same.goreleaser.yml.Likewise the artifact hand-off in commit 1 only exercises on a real push to
main. If the artifact path is wrong the symptom is a failed "Copy release notes from Draft" step, not a malformed PR — it fails closed.Notes for reviewers
--certificate-identity-regexpin the docs assumes the signing job stays inpublish-release.yaml; renaming that file needs the documented command updated too.actions/attest-build-provenance). Both reasonable follow-ups; I kept this to the release artifacts themselves.TestStdioServerMultipleRequestsinpkg/ctl/mcpfailed on ci: sign releases with cosign and publish SBOMs #8861 and passed on ci: keep EKSCTLBOT_TOKEN out of the release-notes rendering job #8862 for the same tree; it looks flaky and is unrelated to these changes (no Go code is touched).