Skip to content

ci: harden the release pipeline — isolate EKSCTLBOT_TOKEN, sign releases, publish SBOMs - #8863

Open
sapphirew wants to merge 2 commits into
eksctl-io:mainfrom
sapphirew:ci/harden-release-pipeline
Open

ci: harden the release pipeline — isolate EKSCTLBOT_TOKEN, sign releases, publish SBOMs#8863
sapphirew wants to merge 2 commits into
eksctl-io:mainfrom
sapphirew:ci/harden-release-pipeline

Conversation

@sapphirew

Copy link
Copy Markdown
Collaborator

Follow-up to #8860. Two independent hardening changes to the release pipeline, kept as two separate commits so either can be dropped without losing the other:

Commit Change
dcf0a94 Splits release-drafter.yaml so the job rendering drafter output no longer holds EKSCTLBOT_TOKEN
d06fe41 Adds cosign keyless signing + per-archive CycloneDX SBOMs

Supersedes #8861 and #8862, which I've closed in favour of this single review.


1. Keep EKSCTLBOT_TOKEN out of the release-notes rendering job

The Draft Release Notes workflow runs as one job that both handles the release-drafter output and holds EKSCTLBOT_TOKEN. Those two things don't need to be in the same job.

EKSCTLBOT_TOKEN is the highest-reach credential in the repo: start-release.yaml triggers on a tag push and passes it to publish-release.yaml, which runs GoReleaser and commits the Homebrew formula to weaveworks/homebrew-tap. Unlike GITHUB_TOKEN, a tag pushed with a PAT does start workflows. Meanwhile the same job does the string handling on steps.draft.outputs.body, which is assembled from merged PR titles — so the least trustworthy input in the workflow sits in the job with the most reach.

#8860 fixed the shell quoting. This separates the two concerns structurally, so the rendering step has nothing valuable within reach even if something about it regresses later.

render_release_draft update_release_draft_pr
permissions contents: write, pull-requests: read contents: read
EKSCTLBOT_TOKEN no yes
checkout none — it doesn't need a working tree yes, persist-credentials: false
drafter output renders to files under RUNNER_TEMP, uploads as artifact consumes as file content only, via body-path

Supporting changes:

  • tag_name promoted to a job output. It comes from tag-template: 'v$NEXT_MINOR_VERSION' in .github/release-drafter.yml and is only used in action inputs (commit-message, title), never in a shell.
  • PR body moves from inline body: to body-path:, rendered with printf. body-path has been available since create-pull-request v6; this repo is on v8.1.1.
  • add-paths: docs/release_notes scopes the commit to the notes file.
  • pull-requests: writeread on the drafter job. .github/release-drafter.yml configures no autolabeler:, so nothing in this workflow applies labels; the PR's own labels are applied by create-pull-request in the second job with its own token. Please sanity-check this one — it's the only permission I reduced based on what the config does rather than what the workflow obviously needs, and it's a one-line revert if I have it wrong.
  • retention-days: 1 on the artifact, since it's only a job-to-job hand-off.
  • if: github.event.repository.fork == false stays on the first job; the second is gated transitively via needs.

2. Sign releases and publish SBOMs

.github/.goreleaser.yml has no signs: or sboms: block, so the only integrity signal on a released binary is eksctl_checksums.txt. That file establishes an archive matches the checksum list, but not that the checksum list came from this pipeline — checksums and archives sit side by side in the same release, so whatever can write one can rewrite the other.

Keyless Sigstore signing over the checksum file. One signature covers the release, because the checksum file already covers every archive. Keyless means the identity is the Actions OIDC token exchanged for a short-lived Fulcio certificate — no long-lived key to store or rotate. Verifiers pin the workflow identity instead:

--certificate-oidc-issuer https://token.actions.githubusercontent.com
--certificate-identity-regexp '^https://github.com/eksctl-io/eksctl/\.github/workflows/publish-release\.yaml@refs/tags/'

This needs id-token: write, added to the publish-release-candidate job in start-release.yaml — the only caller of publish-release.yaml (reusable workflows inherit the calling job's permissions). contents: write and pull-requests: write unchanged.

sboms: uses GoReleaser's defaults, shelling out to syft for <archive>.sbom.json in CycloneDX JSON. cosign and syft are installed as pinned steps before the GoReleaser steps. Both new blocks live in .goreleaser.yml, so the Homebrew release picks them up through the existing cat .goreleaser.yml .goreleaser.brew.yml concatenation. Verification documented in userdocs/src/installation.md.

Checklist

  • Added tests that cover your change (n/a — workflow and release tooling config)
  • Added/modified documentation as required
  • Manually tested
  • Made sure the title of the PR is a good description that can go into the release notes
  • (Core team) Added labels for change area (e.g. area/nodegroup) and target version (e.g. version/0.12.0)

Note

I can't set labels on this repo. The Enforce a valid PR category check needs one of kind/improvement / kind/feature / kind/bug / kind/docs / area/tech-debt / skip-release-noteskind/improvement is the right one. Could a maintainer add it?

Testing

PR body rendering — the part most likely to regress silently, since it moved from a YAML template to printf. Rendered both ways for the same tag and body and diffed, reading the old template straight from git show 8e1d569:.github/workflows/release-drafter.yaml:

IDENTICAL

Also confirmed the notes file lands at docs/release_notes/0.220.0.md for tag v0.220.0, and that a body containing - evil"; whoami; echo " (#124) is written verbatim with nothing executed.

Job graph and permissions:

jobs: ['render_release_draft', 'update_release_draft_pr']
render_release_draft    perms={'contents': 'write', 'pull-requests': 'read'}  needs=None
update_release_draft_pr perms={'contents': 'read'}                           needs=render_release_draft

GoReleaser configgoreleaser wouldn't install in my environment (a transitive dependency fetch is blocked), so I validated against GoReleaser's published JSON schema instead, the same schema goreleaser check uses:

.goreleaser.yml OK
brew combined OK

Both the standalone config and the cat-concatenated Homebrew config validate; artifacts: archive is permitted for sboms and artifacts: checksum for signs per the schema enums.

What I could not verify locally, and would ask a maintainer to confirm on the first tag: that the OIDC exchange succeeds end to end — i.e. id-token: write propagates through the reusable-workflow call and cosign gets its certificate. If it doesn't, the symptom is a GoReleaser failure in the signing stage rather than a silently unsigned release. Worth exercising on an -rc. tag first; the RC path uses the same .goreleaser.yml.

Likewise the artifact hand-off in commit 1 only exercises on a real push to main. If the artifact path is wrong the symptom is a failed "Copy release notes from Draft" step, not a malformed PR — it fails closed.

Notes for reviewers

  • The --certificate-identity-regexp in the docs assumes the signing job stays in publish-release.yaml; renaming that file needs the documented command updated too.
  • The docs say "From v0.220.0 onwards" — please correct to whichever release this actually lands in.
  • Not included: Homebrew/ECR-side signing, and provenance attestations (actions/attest-build-provenance). Both reasonable follow-ups; I kept this to the release artifacts themselves.
  • TestStdioServerMultipleRequests in pkg/ctl/mcp failed on ci: sign releases with cosign and publish SBOMs #8861 and passed on ci: keep EKSCTLBOT_TOKEN out of the release-notes rendering job #8862 for the same tree; it looks flaky and is unrelated to these changes (no Go code is touched).

The Draft Release Notes workflow ran as a single job that both handled
the release-drafter output and held EKSCTLBOT_TOKEN. That token is what
starts the publish pipeline (start-release.yaml triggers on a tag push
and passes it to publish-release.yaml, which runs GoReleaser and commits
the Homebrew formula), so it is the highest-value credential in the repo
and it does not need to be present while the drafter output is being
turned into a file.

Split the job in two:

  render_release_draft   contents: write, pull-requests: read
                         Runs release-drafter, renders the notes file and
                         the pull request body into RUNNER_TEMP, uploads
                         them as an artifact. No EKSCTLBOT_TOKEN, and no
                         checkout -- it does not need the working tree.

  update_release_draft_pr  contents: read
                         Checks out, copies the rendered notes into
                         docs/release_notes/, opens the pull request with
                         EKSCTLBOT_TOKEN. Consumes the drafter output only
                         as file content, via `body-path`, so it is never
                         interpolated into a run step.

Other changes this requires:

- `tag_name` is promoted to a job output. It comes from the
  `tag-template: 'v$NEXT_MINOR_VERSION'` in .github/release-drafter.yml
  and is only used in action inputs, not in a shell.
- The pull request body moves from an inline `body:` to `body-path:`,
  rendered with printf. Verified byte-identical to what the previous YAML
  template produced for the same tag and body.
- `add-paths: docs/release_notes` scopes the commit to the notes file.
- `pull-requests: write` drops to `read` on the drafter job.
  .github/release-drafter.yml configures no `autolabeler:`, so nothing in
  this workflow applies labels; the pull request labels are applied by
  create-pull-request in the second job using its own token.
- `retention-days: 1` on the artifact, since it is only a job-to-job hand-off.

The `if: github.event.repository.fork == false` guard stays on the first
job; the second is gated transitively through `needs`.
`.github/.goreleaser.yml` had no `signs:` or `sboms:` block, so the only
integrity signal on a released eksctl binary was `eksctl_checksums.txt`.
That file establishes that an archive matches the checksum list, but not
that the checksum list itself came from the eksctl release pipeline --
anything that can write to the release can rewrite both.

Add keyless Sigstore signing over the checksum file. Because the checksum
file already covers every archive, one signature covers the whole
release. Keyless means the signing identity is the Actions OIDC token
exchanged for a short-lived Fulcio certificate, so there is no long-lived
signing key to store or rotate; verifiers pin the workflow identity
instead:

  --certificate-oidc-issuer https://token.actions.githubusercontent.com
  --certificate-identity-regexp '^https://github.com/eksctl-io/eksctl/\.github/workflows/publish-release\.yaml@refs/tags/'

This requires `id-token: write`, added to the `publish-release-candidate`
job in start-release.yaml (that job is the only caller of
publish-release.yaml, and reusable workflows inherit the caller's
permissions). `contents: write` and `pull-requests: write` are unchanged.

Also emit a CycloneDX SBOM per archive via syft, so the Go modules
compiled into each binary are enumerable from the release itself.

cosign and syft are installed as pinned steps before the GoReleaser
steps; GoReleaser shells out to both by name. Both new blocks live in
.goreleaser.yml, so the Homebrew release picks them up too via the
existing `cat .goreleaser.yml .goreleaser.brew.yml` concatenation.

Documented verification in userdocs/src/installation.md next to the
existing checksum instructions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant