feat(server): build via BuildKit client when buildkitd address is set

[alexey-igrychev]

Summary

Adds an alternative release-build path that talks to an already running buildkitd through the BuildKit Go client instead of shelling out to docker buildx. The address is set per project via configure (buildkitd_address) or process-wide via TRDL_BUILDKITD_ADDRESS; with no address set, the docker buildx path is unchanged.

Continues #408 by @vmrm — their four commits are taken as-is, with four commits on top that close the findings from reviewing them.

Why

The secret engine is also compiled into host processes shipped in distroless images that contain no docker binary and no docker socket. There exec.CommandContext(ctx, "docker", …) fails with executable file not found before the buildx driver choice from #398 can matter at all. Pointing the plugin at an external buildkitd removes the CLI dependency for unix:///tcp:// while keeping the docker CLI as the default for every existing installation.

The address lives in per-project configure rather than only in the environment because module-based deployments have no way to inject env vars into the Vault pod, and configure is already the per-project channel for s3/git/quorum settings.

Key changes

From #408:

• server/pkg/docker/buildkit.go — the release build mapped onto a single Solve against an external buildkitd: frontend dockerfile.v0, in-context service Dockerfile via filename, no-cache, image-resolve-mode=pull, the context tar streamed through the session upload provider, build secrets and mac-signing credentials served by secretsprovider.FromMap under the ids the generated Dockerfile mounts, tar exporter writing into the same pipe the buildx path writes to, progress through the existing logger.
• server/pkg/docker/builder.go — NewBuilder returns a BuildKit-mode builder when an address resolves: no buildx create, and Remove is a no-op since nothing is provisioned per build.
• server/path_configure.go — optional configure field buildkitd_address, validated at configure time against a scheme allowlist (unix, tcp, docker-container, kube-pod), fail-closed like the feat(server): make buildx driver configurable via env #398 driver allowlist. Per-project config wins over the env fallback.
• Dependencies: github.com/moby/buildkit v0.31.2 direct, golang.org/x/sync promoted from indirect, 46 new modules overall, plugin binary 42.6 MB → 51.5 MB per the measurements in feat(server): build via BuildKit client when buildkitd address is set #408 (not re-measured here). containerd/v2 pinned to v2.2.5 and klauspost/compress to v1.18.7 so the module-level govulncheck advisory set matches main.

On top of that:

• fix(server): the Solve session now also attaches the docker-config auth provider — without it no registry credentials reached buildkitd, so with image-resolve-mode=pull and no cache a private base image could not be resolved at all, while the CLI path gets those credentials through buildx. Builder.Build closes the context reader on return: the upload provider closes it only once buildkitd pulls the context, so a Solve failing earlier left the goroutine streaming the context blocked on write forever with its 64 MiB buffer. An address whose scheme carries no endpoint (unix://) is now rejected at configure time instead of failing on the next release. logWriter returns a wait function, so the tail of a build log is not dropped and the exec path stops leaking its scanner goroutine.
• test(server): the session wiring, the context release and the log drain are asserted against behaviour rather than against the maps the same helpers build. server:test:ai runs everything behind the ai_tests tag, which no task ran before.
• ci: job ai_server starts moby/buildkit:v0.31.2 and points the smoke test at it, so the only test that exercises the Solve path stops being a no-op.
• docs: the buildkitd section states that the build context, the build secrets and the mac-signing credentials travel over that connection, that the client neither encrypts tcp:// nor authenticates the daemon, that the address is a trust boundary for whoever can write configure, and that one daemon is shared by every project pointed at it — all of it the administrator's responsibility.

github.com/docker/cli becomes a direct dependency (already in the graph via the connhelpers) and adds docker/docker-credential-helpers as an indirect one; both were already in go.sum.

Verification

• Live run against buildkitd v0.31.2 over tcp://, using the same recipe the new CI job uses: streamed tar context in, .trdl/Dockerfile picked by filename, a secret mount read back out of the exported artifacts tar (TestAI_BuildkitSmoke, 2s).
• Mutations run against the new tests, each failing only the test that covers it: dropping the auth provider from the session; removing defer contextReader.Close() (the producer test then reports the producer still blocked); removing <-done from the log wait; removing the empty-endpoint check. Before these tests existed, removing the context uploader from the session — which breaks every build — left the whole suite green.
• Not run: the quill-stub mac-signing e2e from feat(server): make buildx driver configurable via env #398 against a buildkitd address; the docker-container:// and kube-pod:// transports live (only tcp:// was exercised); binary size after the docker/cli/cli/config addition.

Review focus / risks

• server/pkg/docker/buildkit.go — the Solve mapping and the session attachables.
• tcp:// is plaintext and unauthenticated: the client has no TLS options, so an operator who ignores the documented requirement ships build secrets and the mac-signing notary key in the clear. Adding mTLS options is deliberately left to a separate change; the current mitigation is documentation only.
• Behaviour differences when, and only when, an address is set: no per-build builder lifecycle, so concurrent releases share one buildkitd and its gc/parallelism limits; secrets are no longer exported into the plugin process environment.
• go.sum churn: MVS bumps of existing indirect deps alongside the new modules.
• Pre-existing and untouched here: the generated Dockerfile mounts certificate_password unconditionally, so a passwordless mac-signing certificate fails in the signing stage on both paths; a build log line above 64 KB kills the bufio.Scanner in logWriter and blocks the release.

After merge

• Close feat(server): build via BuildKit client when buildkitd address is set #408 with a link to this PR — required, the contributor is waiting on a maintainer response there.
• Triage the license/snyk and security/snyk checks; their findings are only visible inside the werf Snyk org.
• File follow-ups for the two pre-existing defects listed above (passwordless mac-signing mount, oversized log line).
• Regenerate docs/_includes/reference/cli/trdl_use.md, stale on main since feat(client): add support for pinning explicit versions in main commands #391 and deliberately left out of this PR.

[vmrm]

Approving. I read the seven commits on top of my four from #408 (73a6cbf9..0aac05e5) — they close real gaps, and I'd rather this PR go in than mine.

What the delta fixes, and why each one matters:

• Registry credentials (buildkitSessionAttachables). The buildx path gets them from the docker config through the CLI; the client path had no auth attachable at all, so image-resolve-mode=pull could only reach public registries. Every private base image would have failed at resolve time, and my e2e never covered that because it pulls public images.
• Draining the build context (defer contextReader.Close() in Builder.Build). A build that fails before the context is fully read left the goroutine feeding the pipe blocked on write forever. This is a leak on the error path of both builders, not just the BuildKit one.
• Log tail (logWriter returning a wait function). Closing the pipe writer did not guarantee the scanner goroutine had drained it, so the last lines of a build log — including the interesting ones on failure — could be dropped. Both call sites now wait.
• Empty endpoint after the scheme. tcp:// alone passed the scheme allowlist and then failed deep inside the client. Rejecting it at configure time is the right place, consistent with the #398 driver allowlist.
• CI against a real buildkitd — End-to-end tests (BuildKit client) running the full release cycle over tcp://, plus the ai_tests_server smoke job. This is what #408 was missing: everything there was verified against unit tests and my own stand, and there was no positive marker in CI that the BuildKit path had actually built anything. With TRDL_TEST_BUILDKITD_ADDRESS threaded into the e2e configure, the same flow-vault suite now covers both backends.
• QUICKSTART security section. Correct, and worth stating explicitly: the plugin ships the whole context and every build secret (including the mac-signing certificate, its password and the notary key) over that connection, tcp:// is plaintext and unauthenticated, and write access to <project>/configure therefore becomes equivalent to access to the release secrets. The daemon-is-shared point is the one people will miss.

All checks are green on 0aac05e5, including the two new jobs.

One nit, not blocking: ValidateBuildkitdAddress and resolveBuildkitdAddress now take a context.Context that neither of them uses — it only ripples out into path_configure.go and the tests. Either use it or drop the parameter; if it's there for a planned reachability check against the address, a // TODO naming that would help.

#408 is closed in favour of this PR.