feat(server): configure buildx driver via plugin configuration - #412
feat(server): configure buildx driver via plugin configuration#412vmrm wants to merge 3 commits into
Conversation
The buildx driver and its --driver-opt values could only be set through the TRDL_BUILDX_DRIVER and TRDL_BUILDX_DRIVER_OPTS_* environment variables of the Vault process. The secret engine is also compiled into host processes whose environment the administrator does not control, and there those variables cannot be set at all, while `configure` is already the per-project channel for the git, s3 and quorum settings. Add the optional `buildx_driver` and `buildx_driver_opts` fields to `configure` and pass them down to `docker buildx create`. Each setting is resolved on its own: the plugin configuration takes precedence over the environment, and the environment over the previous default. With neither field set, the invocation is unchanged. `buildx_driver` is validated against the same allowlist at configure time, so an unsupported driver is rejected when it is written rather than on the next release. `buildx_driver_opts` is a list carrying one option per element, passed through as is, keeping the semantics of the environment form, where a value containing commas is not split. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Vasily Marmer <vasily.marmer@flant.com>
90c7484 to
40199d5
Compare
Both fields are optional, but nothing asserted it: every existing case sends a complete payload, so making either of them required, or giving buildx_driver_opts a non-empty default, would have gone unnoticed until an operator's next configure call failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Vasily Marmer <vasily.marmer@flant.com>
Three gaps found while reviewing the buildx configuration fields. Nothing asserted that the configured driver reaches the builder: dropping the assignment in either BuildReleaseArtifacts or NewBuilder left the whole suite green. Building with an unsupported driver now fails before any docker invocation, which pins the forwarding without a daemon. A configuration stored before these fields existed carries neither key, and nothing decoded such an entry. A rejected update also had no test proving the previously stored configuration survives it, so moving the validation below the write would have gone unnoticed. Also state, in the code and in both QUICKSTART pages, that an empty or omitted field means "not configured" and falls back to the environment rather than overriding it with an empty value: configure cannot tell an omitted field from an explicitly empty one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Vasily Marmer <vasily.marmer@flant.com>
## Summary A project can now choose its buildx driver and driver options through `configure`, instead of only through the environment of the Vault process. That is the only route available when the secret engine is compiled into a host process whose environment the administrator cannot set, where the `kubernetes` driver was unreachable until now. Continues #412 by @vmrm, rebased onto `main` after #409. ## What - `configure` accepts `buildx_driver` — `docker-container` or `kubernetes`, empty by default — and it wins over `TRDL_BUILDX_DRIVER`. - `configure` accepts `buildx_driver_opts` — a list, one `--driver-opt` per element, empty by default — and it wins over `TRDL_BUILDX_DRIVER_OPTS_*`. Elements are passed through verbatim, so `nodeselector=disktype=ssd,zone=a` stays one option. - Each of the two settings resolves on its own: configuration, then environment, then the previous default of `docker-container` with no options. A field omitted or set to a blank value means "not configured" and falls back to the environment instead of clearing it, so building with no options while the environment defines some requires unsetting those variables. - An unsupported driver is rejected whatever its source: writing `buildx_driver=docker` fails `configure` and stores nothing, while an unsupported `TRDL_BUILDX_DRIVER` still fails when the builder is created. The error names the setting the value came from. - `configure` rejects `buildx_driver` or `buildx_driver_opts` written together with `buildkitd_address`, and leaves an already stored configuration untouched: that address replaces the buildx path entirely, so the driver settings would have no effect. Blank values count as unset on both sides of that check. - When the address comes from `TRDL_BUILDKITD_ADDRESS` instead, the combination cannot be refused at write time — the variable is process-wide and can change after a project is configured — so the release reports `the configured buildx driver settings are not used` in both the release task log and the plugin log. - An update that omits the buildx fields clears the stored ones: `configure` replaces the whole document. - With neither field set, `docker buildx create` is invoked exactly as before, down to the argument list, and a configuration stored before these fields existed reads back with both fields empty and builds as it did. - UNVERIFIED: that `buildx_driver_opts` and `buildkitd_address` reach the release build at all — deleting their assignment in `pathRelease` leaves every suite green. The driver now has an end-to-end guard; these two would need one job each, because the value has to break the build when it is lost. ## Why Everything else a project needs already lives in `configure` — the Git repository, the S3 credentials, the signature quorum — because Vault has no other per-project channel. The build backend was the exception: it could only be set through the environment of the process that hosts the plugin, and a host process that ships the engine as a built-in plugin may expose no way to set one, which left such an installation on the default driver permanently. Environment-only was the alternative, and #409 shows why it does not hold: the same argument produced `buildkitd_address` as a `configure` field, so keeping the driver out of `configure` would have split one decision across two mechanisms.
|
Merged as #419 — thank you. Your three commits are its base, rebased onto Your What changed on top of your work:
One thing worth knowing for future tests, because it cost real resources here. Also from this pair of PRs: #410 and #411 were pre-existing defects found while reviewing them, both fixed in #416, and #413 proposes dropping the |
Summary
Make the buildx driver from #398 settable per project through
configure, not only through the environment of the Vault process. Two optional fields,buildx_driverandbuildx_driver_opts; with neither set,docker buildx createis invoked exactly as before.Key changes
server/path_configure.go: two optionalconfigurefields alongsidegit_*/s3_*:buildx_driver(string) — same values asTRDL_BUILDX_DRIVER, validated at configure time against the same allowlist (docker-container,kubernetes), so an unsupported driver is rejected when it is written rather than on the next release;buildx_driver_opts(list of strings) — one--driver-optper element, each passed through as is. A list rather than a delimited string keeps the fix(server): stop splitting buildx driver opts by default #399 semantics:nodeselector=disktype=ssd,zone=ais one option and is not split. TheTRDL_BUILDX_DRIVER_OPTS_SEPARATORknob has no configuration counterpart — it exists only to pack several options into one variable, which a list does not need.server/pkg/docker/builder.go:buildxCreateArgstakes the driver and the options as arguments;resolveBuildxDriver/resolveBuildxDriverOptsresolve each setting independently — plugin configuration, then environment, then the previous default (docker-container, no options).ValidateBuildxDriveris exported for the configure-time check and treats an empty driver as "not set". The rejection message names the setting the value came from.server/pkg/docker/build.go,server/path_release.go: the two values are threaded from the stored configuration intoNewBuilderOpts, next to the existing per-project settings.configureAPI partial is regenerated (task docs:genregenerates the CLI partials too, which are stale onmainfor unrelated reasons — only the vault-plugin partial is included here).server/pkg/docker/builder_ai_test.go,server/pkg/docker/build_ai_test.go,server/path_configure_ai_test.go(perAGENTS.md, agent-written tests are*_ai_test.gobehind theai_teststag). The existingcompleteConfiguration()fixture now carries both fields, so the pre-existing configure round-trip test covers them — including a comma-containing option value, which is the regression guard for the Vault field layer not splitting it.Why
The trdl secret engine is also compiled into host processes that ship it as a built-in plugin. There,
plugin enable -envdoes not apply, and the surrounding deployment may expose no way to set environment variables on the process at all — the settings schema simply has no field for them. In that shape everything else the plugin needs is already reachable (git_repo_url,s3_*, the signature quorum) because it lives inconfigure, while the build backend, alone, is reachable only through the environment. Putting the driver where the rest of the per-project configuration lives removes that inconsistency and is the only way to select thekubernetesdriver in such an installation.Follow-up to #398 (env-configurable driver) and #399 (options passed through, not split).
Review focus / risks
Backwards compatibility. Both fields are optional and default to the zero value;
TestBuildxCreateArgs_DefaultDriverUnchangedand the rest of the pre-existingTestBuildxCreateArgs_*suite still pin the byte-for-bytebuildx createinvocation, and they now run with the new parameters explicitly empty. A configuration stored before these fields existed decodes to empty values and keeps the previous behaviour (TestAI_Read_ConfigurationStoredBeforeBuildxFields).An empty value means "not configured", not "override with nothing".
configurecannot distinguish an omitted field from an explicitly empty one — the Vault field layer returns""/[]string{}for both — so an emptybuildx_driver_optsfalls back toTRDL_BUILDX_DRIVER_OPTS_*rather than clearing it. Building with no driver options while the environment defines some therefore requires unsetting those variables. Stated inresolveBuildxDriverOptsand in both QUICKSTART pages.Precedence is per setting, not per group: configuring only
buildx_driverleavesTRDL_BUILDX_DRIVER_OPTS_*in effect. This is deliberate — an operator who adds the driver toconfigureshould not silently lose the options they already set in the environment — but it is the one behaviour here that could surprise.TestAI_BuildxCreateArgs_ConfiguredDriverKeepsEnvOptspins it.Coverage of the forwarding chain.
BuildReleaseArtifactsOpts→NewBuilderOpts→buildxCreateArgsis covered byTestAI_BuildReleaseArtifacts_ForwardsConfiguredDriverandTestAI_NewBuilder_UsesConfiguredDriver, both of which fail before anydockerinvocation, so they need no daemon. The remaining hop —path_release.goreadingcfgintoBuildReleaseArtifactsOpts— is still uncovered; its entry point is the release task, behind a git clone and PGP quorum verification.The new tests do not run in the required checks.
AGENTS.mdmandates theai_testsbuild tag for agent-written tests, and no task onmainpasses it, sounit_servercompiles them and runs none. I deliberately did not add a second runner here: feat(server): build via BuildKit client when buildkitd address is set #409 already introducesserver:test:aiand a CI job for exactly this tag, and duplicating it would collide with that PR. Until one of them lands, these tests are run withginkgo --vet=off --tags=ai_tests ./server/ ./server/pkg/docker/. What the required checks do cover is the part that matters for existing installations: the unchanged default invocation and the configure round-trip, both in untagged tests.Verified locally (darwin/arm64):
task server:test:unit— 9 suites green; theai_testsrun above — green;task server:lint -- --build-tags=ai_tests,task docs:lint:prettier— clean.Mutation-tested (each fault applied to the implementation, suite re-run, fault reverted) — every one killed, by the test that covers it:
TestAI_BuildxCreateArgs_ConfigurationOverridesEnvTestAI_BuildxCreateArgs_ConfigurationOverridesEnv,…_ConfiguredOptsPassedThroughAndTrimmedTestAI_BuildxCreateArgs_EnvUsedWhenConfigurationEmpty,…_ConfiguredDriverKeepsEnvOpts(+ 5 pre-existing),(the fix(server): stop splitting buildx driver opts by default #399 bug)TestAI_BuildxCreateArgs_ConfiguredOptsPassedThroughAndTrimmedNewBuilderdrops the configured driverTestAI_NewBuilder_UsesConfiguredDriverBuildReleaseArtifactsdrops the configured driverTestAI_BuildReleaseArtifacts_ForwardsConfiguredDriverTestAI_CreateOrUpdate_UnsupportedBuildxDriver,…_RejectedUpdateKeepsConfigurationTestAI_CreateOrUpdate_RejectedUpdateKeepsConfiguration,…_UnsupportedBuildxDriverbuildx_drivermade requiredTestAI_CreateOrUpdate_BuildxFieldsOmittedbuildx_driver_optsgiven a non-empty defaultTestAI_CreateOrUpdate_BuildxFieldsOmitted,TestCreateOrUpdate_CompleteConfigurationconfiguredropsbuildx_driver_optsTestCreateOrUpdate_CompleteConfigurationNot mutated:
TestAI_Read_ConfigurationStoredBeforeBuildxFields— the smallest fault to try against it is makinggetConfigurationreject an entry whosebuildx_driver_optskey is absent.Overlap with feat(server): build via BuildKit client when buildkitd address is set #408/feat(server): build via BuildKit client when buildkitd address is set #409: those add a
buildkitd_addressfield to the sameconfigurepath and touchbuilder.go/build.go/path_release.go. This PR is independent of them and branches offmain; the overlaps are adjacent-line only. If feat(server): build via BuildKit client when buildkitd address is set #409 lands first I will rebase.🤖 Generated with Claude Code