Skip to content

feat(server): configure buildx driver via plugin configuration - #412

Closed
vmrm wants to merge 3 commits into
werf:mainfrom
vmrm:feat/server/buildx-driver-in-plugin-config
Closed

feat(server): configure buildx driver via plugin configuration#412
vmrm wants to merge 3 commits into
werf:mainfrom
vmrm:feat/server/buildx-driver-in-plugin-config

Conversation

@vmrm

@vmrm vmrm commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Make the buildx driver from #398 settable per project through configure, not only through the environment of the Vault process. Two optional fields, buildx_driver and buildx_driver_opts; with neither set, docker buildx create is invoked exactly as before.

Key changes

  • server/path_configure.go: two optional configure fields alongside git_*/s3_*:
    • buildx_driver (string) — same values as TRDL_BUILDX_DRIVER, validated at configure time against the same allowlist (docker-container, kubernetes), so an unsupported driver is rejected when it is written rather than on the next release;
    • buildx_driver_opts (list of strings) — one --driver-opt per element, each passed through as is. A list rather than a delimited string keeps the fix(server): stop splitting buildx driver opts by default #399 semantics: nodeselector=disktype=ssd,zone=a is one option and is not split. The TRDL_BUILDX_DRIVER_OPTS_SEPARATOR knob has no configuration counterpart — it exists only to pack several options into one variable, which a list does not need.
  • server/pkg/docker/builder.go: buildxCreateArgs takes the driver and the options as arguments; resolveBuildxDriver / resolveBuildxDriverOpts resolve each setting independently — plugin configuration, then environment, then the previous default (docker-container, no options). ValidateBuildxDriver is exported for the configure-time check and treats an empty driver as "not set". The rejection message names the setting the value came from.
  • server/pkg/docker/build.go, server/path_release.go: the two values are threaded from the stored configuration into NewBuilderOpts, next to the existing per-project settings.
  • Docs: QUICKSTART (en/ru) "Build backend" section gains the configuration form and states the precedence; the generated configure API partial is regenerated (task docs:gen regenerates the CLI partials too, which are stale on main for unrelated reasons — only the vault-plugin partial is included here).
  • Tests: server/pkg/docker/builder_ai_test.go, server/pkg/docker/build_ai_test.go, server/path_configure_ai_test.go (per AGENTS.md, agent-written tests are *_ai_test.go behind the ai_tests tag). The existing completeConfiguration() fixture now carries both fields, so the pre-existing configure round-trip test covers them — including a comma-containing option value, which is the regression guard for the Vault field layer not splitting it.

Why

The trdl secret engine is also compiled into host processes that ship it as a built-in plugin. There, plugin enable -env does not apply, and the surrounding deployment may expose no way to set environment variables on the process at all — the settings schema simply has no field for them. In that shape everything else the plugin needs is already reachable (git_repo_url, s3_*, the signature quorum) because it lives in configure, while the build backend, alone, is reachable only through the environment. Putting the driver where the rest of the per-project configuration lives removes that inconsistency and is the only way to select the kubernetes driver in such an installation.

Follow-up to #398 (env-configurable driver) and #399 (options passed through, not split).

Review focus / risks

  • Backwards compatibility. Both fields are optional and default to the zero value; TestBuildxCreateArgs_DefaultDriverUnchanged and the rest of the pre-existing TestBuildxCreateArgs_* suite still pin the byte-for-byte buildx create invocation, and they now run with the new parameters explicitly empty. A configuration stored before these fields existed decodes to empty values and keeps the previous behaviour (TestAI_Read_ConfigurationStoredBeforeBuildxFields).

  • An empty value means "not configured", not "override with nothing". configure cannot distinguish an omitted field from an explicitly empty one — the Vault field layer returns "" / []string{} for both — so an empty buildx_driver_opts falls back to TRDL_BUILDX_DRIVER_OPTS_* rather than clearing it. Building with no driver options while the environment defines some therefore requires unsetting those variables. Stated in resolveBuildxDriverOpts and in both QUICKSTART pages.

  • Precedence is per setting, not per group: configuring only buildx_driver leaves TRDL_BUILDX_DRIVER_OPTS_* in effect. This is deliberate — an operator who adds the driver to configure should not silently lose the options they already set in the environment — but it is the one behaviour here that could surprise. TestAI_BuildxCreateArgs_ConfiguredDriverKeepsEnvOpts pins it.

  • Coverage of the forwarding chain. BuildReleaseArtifactsOptsNewBuilderOptsbuildxCreateArgs is covered by TestAI_BuildReleaseArtifacts_ForwardsConfiguredDriver and TestAI_NewBuilder_UsesConfiguredDriver, both of which fail before any docker invocation, so they need no daemon. The remaining hop — path_release.go reading cfg into BuildReleaseArtifactsOpts — is still uncovered; its entry point is the release task, behind a git clone and PGP quorum verification.

  • The new tests do not run in the required checks. AGENTS.md mandates the ai_tests build tag for agent-written tests, and no task on main passes it, so unit_server compiles them and runs none. I deliberately did not add a second runner here: feat(server): build via BuildKit client when buildkitd address is set #409 already introduces server:test:ai and a CI job for exactly this tag, and duplicating it would collide with that PR. Until one of them lands, these tests are run with ginkgo --vet=off --tags=ai_tests ./server/ ./server/pkg/docker/. What the required checks do cover is the part that matters for existing installations: the unchanged default invocation and the configure round-trip, both in untagged tests.

  • Verified locally (darwin/arm64): task server:test:unit — 9 suites green; the ai_tests run above — green; task server:lint -- --build-tags=ai_tests, task docs:lint:prettier — clean.

  • Mutation-tested (each fault applied to the implementation, suite re-run, fault reverted) — every one killed, by the test that covers it:

    Mutation Failing test
    environment wins over the configuration (driver) TestAI_BuildxCreateArgs_ConfigurationOverridesEnv
    configured options ignored TestAI_BuildxCreateArgs_ConfigurationOverridesEnv, …_ConfiguredOptsPassedThroughAndTrimmed
    environment option fallback dropped TestAI_BuildxCreateArgs_EnvUsedWhenConfigurationEmpty, …_ConfiguredDriverKeepsEnvOpts (+ 5 pre-existing)
    configured options split on , (the fix(server): stop splitting buildx driver opts by default #399 bug) TestAI_BuildxCreateArgs_ConfiguredOptsPassedThroughAndTrimmed
    NewBuilder drops the configured driver TestAI_NewBuilder_UsesConfiguredDriver
    BuildReleaseArtifacts drops the configured driver TestAI_BuildReleaseArtifacts_ForwardsConfiguredDriver
    configure-time validation neutered TestAI_CreateOrUpdate_UnsupportedBuildxDriver, …_RejectedUpdateKeepsConfiguration
    validation moved below the write TestAI_CreateOrUpdate_RejectedUpdateKeepsConfiguration, …_UnsupportedBuildxDriver
    buildx_driver made required TestAI_CreateOrUpdate_BuildxFieldsOmitted
    buildx_driver_opts given a non-empty default TestAI_CreateOrUpdate_BuildxFieldsOmitted, TestCreateOrUpdate_CompleteConfiguration
    configure drops buildx_driver_opts TestCreateOrUpdate_CompleteConfiguration

    Not mutated: TestAI_Read_ConfigurationStoredBeforeBuildxFields — the smallest fault to try against it is making getConfiguration reject an entry whose buildx_driver_opts key is absent.

  • Overlap with feat(server): build via BuildKit client when buildkitd address is set #408/feat(server): build via BuildKit client when buildkitd address is set #409: those add a buildkitd_address field to the same configure path and touch builder.go/build.go/path_release.go. This PR is independent of them and branches off main; the overlaps are adjacent-line only. If feat(server): build via BuildKit client when buildkitd address is set #409 lands first I will rebase.

🤖 Generated with Claude Code

The buildx driver and its --driver-opt values could only be set through
the TRDL_BUILDX_DRIVER and TRDL_BUILDX_DRIVER_OPTS_* environment
variables of the Vault process. The secret engine is also compiled into
host processes whose environment the administrator does not control, and
there those variables cannot be set at all, while `configure` is already
the per-project channel for the git, s3 and quorum settings.

Add the optional `buildx_driver` and `buildx_driver_opts` fields to
`configure` and pass them down to `docker buildx create`. Each setting is
resolved on its own: the plugin configuration takes precedence over the
environment, and the environment over the previous default. With neither
field set, the invocation is unchanged.

`buildx_driver` is validated against the same allowlist at configure
time, so an unsupported driver is rejected when it is written rather
than on the next release. `buildx_driver_opts` is a list carrying one
option per element, passed through as is, keeping the semantics of the
environment form, where a value containing commas is not split.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Vasily Marmer <vasily.marmer@flant.com>
@vmrm
vmrm force-pushed the feat/server/buildx-driver-in-plugin-config branch from 90c7484 to 40199d5 Compare August 5, 2026 16:12
@vmrm
vmrm marked this pull request as ready for review August 5, 2026 16:22
vmrm and others added 2 commits August 5, 2026 18:46
Both fields are optional, but nothing asserted it: every existing case
sends a complete payload, so making either of them required, or giving
buildx_driver_opts a non-empty default, would have gone unnoticed until
an operator's next configure call failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Vasily Marmer <vasily.marmer@flant.com>
Three gaps found while reviewing the buildx configuration fields.

Nothing asserted that the configured driver reaches the builder: dropping
the assignment in either BuildReleaseArtifacts or NewBuilder left the
whole suite green. Building with an unsupported driver now fails before
any docker invocation, which pins the forwarding without a daemon.

A configuration stored before these fields existed carries neither key,
and nothing decoded such an entry. A rejected update also had no test
proving the previously stored configuration survives it, so moving the
validation below the write would have gone unnoticed.

Also state, in the code and in both QUICKSTART pages, that an empty or
omitted field means "not configured" and falls back to the environment
rather than overriding it with an empty value: configure cannot tell an
omitted field from an explicitly empty one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Vasily Marmer <vasily.marmer@flant.com>
alexey-igrychev added a commit that referenced this pull request Aug 5, 2026
## Summary

A project can now choose its buildx driver and driver options through
`configure`, instead of only through the environment of the Vault
process. That is the only route available when the secret engine is
compiled into a host process whose environment the administrator cannot
set, where the `kubernetes` driver was unreachable until now.

Continues #412 by @vmrm, rebased onto `main` after #409.

## What

- `configure` accepts `buildx_driver` — `docker-container` or
`kubernetes`, empty by default — and it wins over `TRDL_BUILDX_DRIVER`.
- `configure` accepts `buildx_driver_opts` — a list, one `--driver-opt`
per element, empty by default — and it wins over
`TRDL_BUILDX_DRIVER_OPTS_*`. Elements are passed through verbatim, so
`nodeselector=disktype=ssd,zone=a` stays one option.
- Each of the two settings resolves on its own: configuration, then
environment, then the previous default of `docker-container` with no
options. A field omitted or set to a blank value means "not configured"
and falls back to the environment instead of clearing it, so building
with no options while the environment defines some requires unsetting
those variables.
- An unsupported driver is rejected whatever its source: writing
`buildx_driver=docker` fails `configure` and stores nothing, while an
unsupported `TRDL_BUILDX_DRIVER` still fails when the builder is
created. The error names the setting the value came from.
- `configure` rejects `buildx_driver` or `buildx_driver_opts` written
together with `buildkitd_address`, and leaves an already stored
configuration untouched: that address replaces the buildx path entirely,
so the driver settings would have no effect. Blank values count as unset
on both sides of that check.
- When the address comes from `TRDL_BUILDKITD_ADDRESS` instead, the
combination cannot be refused at write time — the variable is
process-wide and can change after a project is configured — so the
release reports `the configured buildx driver settings are not used` in
both the release task log and the plugin log.
- An update that omits the buildx fields clears the stored ones:
`configure` replaces the whole document.
- With neither field set, `docker buildx create` is invoked exactly as
before, down to the argument list, and a configuration stored before
these fields existed reads back with both fields empty and builds as it
did.
- UNVERIFIED: that `buildx_driver_opts` and `buildkitd_address` reach
the release build at all — deleting their assignment in `pathRelease`
leaves every suite green. The driver now has an end-to-end guard; these
two would need one job each, because the value has to break the build
when it is lost.

## Why

Everything else a project needs already lives in `configure` — the Git
repository, the S3 credentials, the signature quorum — because Vault has
no other per-project channel. The build backend was the exception: it
could only be set through the environment of the process that hosts the
plugin, and a host process that ships the engine as a built-in plugin
may expose no way to set one, which left such an installation on the
default driver permanently.

Environment-only was the alternative, and #409 shows why it does not
hold: the same argument produced `buildkitd_address` as a `configure`
field, so keeping the driver out of `configure` would have split one
decision across two mechanisms.
@alexey-igrychev

Copy link
Copy Markdown
Member

Merged as #419 — thank you. Your three commits are its base, rebased onto main after #409 as you offered; the squash collapsed everything into c68e805, so main names this PR and you in the commit body rather than carrying your commits individually.

Your TypeStringSlice choice was the right one and I verified it end to end rather than by reading the type: nodeselector=disktype=ssd,zone=a submitted as a vault write flag, as an HTTP JSON array and as a Go []string all store as exactly one element, parseDriverOpts(opt, "") never splits, and the final --driver-opt=… is a single argv element. The regression guard you built into the fixture round trip does bite — splitting stored options on commas fails TestCreateOrUpdate_CompleteConfiguration.

What changed on top of your work:

  • configure now rejects the buildx fields written together with buildkitd_address. That address replaces the whole buildx path, so a driver next to it would silently do nothing; it is refused at write time, and an already stored configuration is left untouched. Blank values count as unset on both sides, matching how resolution treats them.
  • The unreachable case is reported instead. When the address comes from TRDL_BUILDKITD_ADDRESS the combination cannot be refused at write time — the variable is process-wide and can change after a project is configured — so the build states that the configured driver settings are unused, in both the release task log and the plugin log.
  • ValidateBuildxDriver takes a context, which AGENTS.md requires of an exported function.
  • The pathRelease forwarding is guarded. Deleting the two lines that carry the stored settings into the build left every suite green, so a new e2e job configures the working driver through configure while the environment names one that cannot work without a cluster: the release only succeeds if the value travels the whole way.
  • The completeConfiguration fixture dropped buildkitd_address, since after the rebase it described a configuration the new check rejects; the buildx pair kept its round trip, the address has its own tests.

One thing worth knowing for future tests, because it cost real resources here. TestAI_BuildReleaseArtifacts_ForwardsConfiguredDriver and TestAI_NewBuilder_UsesConfiguredDriver relied on driver validation stopping the build before docker buildx create — but that barrier lives inside the code being mutated. A reviewer inverted the resolution precedence as a routine mutation, and the first test then ran a real docker buildx create --driver=kubernetes, booted a BuildKit Deployment in their cluster, pulled alpine, ran the release Dockerfile, and deadlocked the test binary on an artifacts pipe nobody read — printing no --- FAIL: line at all, and leaving the builder behind because Builder.Remove was never reached. Both now run with an empty PATH and drain the pipe, so that fault fails the test instead of provisioning infrastructure. The general rule went into .agents/skills/test-the-tests in #420.

Also from this pair of PRs: #410 and #411 were pre-existing defects found while reviewing them, both fixed in #416, and #413 proposes dropping the ai_tests convention that keeps tests like yours out of the required checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants