Skip to content

fix(server): unhang the release on a long log line and a passwordless certificate - #416

Merged
alexey-igrychev merged 2 commits into
mainfrom
fix/server/release-build-defects
Aug 5, 2026
Merged

fix(server): unhang the release on a long log line and a passwordless certificate#416
alexey-igrychev merged 2 commits into
mainfrom
fix/server/release-build-defects

Conversation

@alexey-igrychev

Copy link
Copy Markdown
Member

Summary

Two release-build defects that predate #409 and affect both build paths: an oversized build log line hangs the release task, and a passwordless mac signing certificate fails the signer stage.

Why

Both turn an ordinary situation into a broken release, and neither reports a cause the operator can act on.

bufio.Scanner stops at bufio.ErrTooLong on a line above 64 KB, after which the goroutine reading the pipe returns. The build keeps writing into that pipe, so the next write blocks forever — cmd.Run on the docker CLI path, the progress display on the BuildKit client path — and the release task hangs with no error, until the plugin process is restarted.

The generated service Dockerfile mounts certificate_password unconditionally and reads it with cat under set -e -o pipefail, while both build paths served that secret only when the stored password was non-empty. --mount=type=secret defaults to required=false, so BuildKit mounts nothing and the cat takes the whole RUN down. A passwordless p12 is legitimate.

Key changes

  • server/pkg/docker/builder.gologWriter raises the line limit to 1 MiB and drains the pipe to io.Discard in a deferred call, so parsing that stops for any reason can no longer block the writer. An oversized line now costs the rest of the build log, reported through the logger, instead of the release.
  • server/pkg/docker/mac_signing.go, server/pkg/docker/buildkit.go — the password secret is always served, with an empty value when there is none, so quill receives an empty QUILL_SIGN_PASSWORD. Both paths mount the same ids and are changed together.
  • server/pkg/docker/buildkit_test.goTestBuildkitSecretsData_NoPasswordNoCredentials asserted the defect (NotContains the password id); the passwordless half is replaced by the new test below, and what remains covers the no-credentials case.

Verification

  • Mutations run, each reverted after the run, each killing exactly the test that covers it:
    • removing the drain from logWriterTestAI_LogWriter_OversizedLineDoesNotBlockTheBuild fails after its 30s timeout, reporting the blocked writer;
    • restoring if Password != "" in buildkitSecretsDataTestAI_MacSigningSecrets_PasswordlessCertificateIsStillServed fails;
    • restoring the same condition in GetMacSigningCommandMounts → the same test fails, so both halves of it discriminate.
  • Not run: a release with a passwordless certificate through the mac-signing e2e suite. Its quill stub validates the five QUILL_* variables, so accepting an empty password there is a change to the fixture rather than a run of the existing one.

Review focus / risks

  • The 1 MiB limit is a judgement call: below it a line is buffered whole, above it the remaining log is dropped. The hang is gone either way, and the drop is logged.
  • An empty QUILL_SIGN_PASSWORD reaches quill for a passwordless certificate. This is the intended reading of "no password", but it is a behaviour change for anyone who stored an empty password expecting the release to fail.

After merge

A single build log line above 64 KB made `bufio.Scanner` stop with
`bufio.ErrTooLong`, and the goroutine reading the pipe returned. Nobody was
reading the pipe the build writes into any more, so the next write blocked
forever: on the docker CLI path inside `cmd.Run`, on the BuildKit client path
inside the progress display, and the release task then hung with no error at
all.

Raise the line limit to 1 MiB and, whatever the parsing does, keep draining the
pipe until the writer closes it. An oversized line now costs the rest of the
build log, reported through the logger, instead of the release.

Fixes #411

Signed-off-by: Aleksei Igrychev <aleksei.igrychev@palark.com>
The generated service Dockerfile mounts `certificate_password` unconditionally
and reads it with `cat` under `set -e -o pipefail`, while both build paths
provided that secret only when the stored password was non-empty. A
passwordless p12 is legitimate, but it took the signer stage down on a missing
`/run/secrets/certificate_password`, with nothing in the output pointing at the
password as the cause.

Always serve the secret and let its value be empty, so quill receives an empty
`QUILL_SIGN_PASSWORD`. The exec path and the BuildKit client path are changed
together, since they mount the same ids.

Fixes #410

Signed-off-by: Aleksei Igrychev <aleksei.igrychev@palark.com>
@alexey-igrychev
alexey-igrychev marked this pull request as ready for review August 5, 2026 19:09
@alexey-igrychev
alexey-igrychev merged commit 82fcfa0 into main Aug 5, 2026
26 checks passed
@alexey-igrychev
alexey-igrychev deleted the fix/server/release-build-defects branch August 5, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release fails when the mac signing certificate has no password

1 participant