fix(server): unhang the release on a long log line and a passwordless certificate - #416
Merged
Merged
Conversation
A single build log line above 64 KB made `bufio.Scanner` stop with `bufio.ErrTooLong`, and the goroutine reading the pipe returned. Nobody was reading the pipe the build writes into any more, so the next write blocked forever: on the docker CLI path inside `cmd.Run`, on the BuildKit client path inside the progress display, and the release task then hung with no error at all. Raise the line limit to 1 MiB and, whatever the parsing does, keep draining the pipe until the writer closes it. An oversized line now costs the rest of the build log, reported through the logger, instead of the release. Fixes #411 Signed-off-by: Aleksei Igrychev <aleksei.igrychev@palark.com>
The generated service Dockerfile mounts `certificate_password` unconditionally and reads it with `cat` under `set -e -o pipefail`, while both build paths provided that secret only when the stored password was non-empty. A passwordless p12 is legitimate, but it took the signer stage down on a missing `/run/secrets/certificate_password`, with nothing in the output pointing at the password as the cause. Always serve the secret and let its value be empty, so quill receives an empty `QUILL_SIGN_PASSWORD`. The exec path and the BuildKit client path are changed together, since they mount the same ids. Fixes #410 Signed-off-by: Aleksei Igrychev <aleksei.igrychev@palark.com>
alexey-igrychev
marked this pull request as ready for review
August 5, 2026 19:09
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two release-build defects that predate #409 and affect both build paths: an oversized build log line hangs the release task, and a passwordless mac signing certificate fails the signer stage.
Why
Both turn an ordinary situation into a broken release, and neither reports a cause the operator can act on.
bufio.Scannerstops atbufio.ErrTooLongon a line above 64 KB, after which the goroutine reading the pipe returns. The build keeps writing into that pipe, so the next write blocks forever —cmd.Runon the docker CLI path, the progress display on the BuildKit client path — and the release task hangs with no error, until the plugin process is restarted.The generated service Dockerfile mounts
certificate_passwordunconditionally and reads it withcatunderset -e -o pipefail, while both build paths served that secret only when the stored password was non-empty.--mount=type=secretdefaults torequired=false, so BuildKit mounts nothing and thecattakes the wholeRUNdown. A passwordless p12 is legitimate.Key changes
server/pkg/docker/builder.go—logWriterraises the line limit to 1 MiB and drains the pipe toio.Discardin a deferred call, so parsing that stops for any reason can no longer block the writer. An oversized line now costs the rest of the build log, reported through the logger, instead of the release.server/pkg/docker/mac_signing.go,server/pkg/docker/buildkit.go— the password secret is always served, with an empty value when there is none, so quill receives an emptyQUILL_SIGN_PASSWORD. Both paths mount the same ids and are changed together.server/pkg/docker/buildkit_test.go—TestBuildkitSecretsData_NoPasswordNoCredentialsasserted the defect (NotContainsthe password id); the passwordless half is replaced by the new test below, and what remains covers the no-credentials case.Verification
logWriter→TestAI_LogWriter_OversizedLineDoesNotBlockTheBuildfails after its 30s timeout, reporting the blocked writer;if Password != ""inbuildkitSecretsData→TestAI_MacSigningSecrets_PasswordlessCertificateIsStillServedfails;GetMacSigningCommandMounts→ the same test fails, so both halves of it discriminate.QUILL_*variables, so accepting an empty password there is a change to the fixture rather than a run of the existing one.Review focus / risks
QUILL_SIGN_PASSWORDreaches quill for a passwordless certificate. This is the intended reading of "no password", but it is a behaviour change for anyone who stored an empty password expecting the release to fail.After merge
Fixesin the commits, so a squash merge closes them automatically — verify it did).