solver: add per-step CPU and memory resource limits by jirimoravcik · Pull Request #6569 · moby/buildkit

jirimoravcik · 2026-03-10T15:50:14Z

Adds support for setting cgroup resource limits on individual build steps: memory, memory-swap, cpu-shares, cpu-period, cpu-quota, cpuset-cpus, and cpuset-mems. These correspond to the legacy Docker build API. In case of several builds with different limits, the most relaxed limits are chosen.

The limits do not affect the cache key.

I think we should document that the BuildKit steps can run in parallel anyway and mention that these limits only apply to a single step. (On the other hand, I still feel this can be useful)

Closes #593

Copilot

Pull request overview

Adds support for specifying per-exec Linux cgroup CPU/memory constraints via LLB OpMetadata (so limits affect runtime execution but not the cache key), plumbed from the Dockerfile frontend/client through the solver into OCI spec generation.

Changes:

Introduces LinuxResources into pb.OpMetadata and propagates it through vertex options, exec op handling, and executor metadata.
Adds client-side LLB APIs (WithLinuxResources, MemoryLimit, CPUQuota, etc.) and Dockerfile/frontend attribute parsing to set these limits.
Extends OCI spec generation (Linux-only) to apply the resource limits, with integration/unit tests covering behavior and cache-key invariance.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
solver/types.go	Extends `VertexOptions` to carry Linux resource constraints.
solver/pb/ops.proto	Adds `LinuxResources` message + `linux_resources` field to `OpMetadata`.
solver/pb/ops.pb.go	Regenerates protobuf Go bindings for new metadata field/message.
solver/pb/ops_vtproto.pb.go	Regenerates vtproto fast-paths to include `LinuxResources`.
solver/pb/caps.go	Introduces and registers capability `exec.meta.linux.resources`.
solver/llbsolver/vertex.go	Propagates `OpMetadata.LinuxResources` into vertex options.
solver/llbsolver/ops/exec.go	Threads linux resource limits into `executor.Meta` for exec.
frontend/dockerui/config.go	Adds new frontend attr keys + stores parsed linux resources in config.
frontend/dockerui/attr.go	Implements parsing of linux resource attrs into `pb.LinuxResources`.
frontend/dockerfile/dockerfile2llb/convert.go	Applies linux resource constraints to `RUN` when cap supported.
frontend/dockerfile/dockerfile_test.go	Adds integration test verifying cgroup memory limit from Dockerfile attrs.
executor/executor.go	Extends `executor.Meta` to include `LinuxResources`.
executor/oci/spec.go	Hooks linux resource option generation into spec creation pipeline.
executor/oci/spec_linux.go	Implements OCI spec mutations for CPU/memory/cpuset limits on Linux.
executor/oci/spec_windows.go	Returns a clear error for linux resource limits on Windows.
executor/oci/spec_freebsd.go	Returns a clear error for linux resource limits on FreeBSD.
executor/oci/spec_darwin.go	Returns a clear error for linux resource limits on Darwin.
client/llb/meta.go	Adds public `llb.LinuxResources` struct for the client API.
client/llb/state.go	Adds metadata plumbing and constraint opts for linux resources.
client/llb/exec.go	Emits capability when linux resources are present in metadata.
client/llb/exec_test.go	Adds unit tests for metadata presence, merge behavior, and cache-key invariance.
client/client_test.go	Adds integration test validating cgroup limits during solves.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tonistiigi · 2026-03-10T16:08:32Z

The limits do not affect the cache key.

I think we should document that the BuildKit steps can run in parallel anyway and mention that these limits only apply to a single step. (On the other hand, I still feel this can be useful)

Do I understand correctly that when same step has different resource definitions then what is actually used for the step that runs is undefined?

jirimoravcik · 2026-03-10T21:48:16Z

The limits do not affect the cache key.

I think we should document that the BuildKit steps can run in parallel anyway and mention that these limits only apply to a single step. (On the other hand, I still feel this can be useful)

Do I understand correctly that when same step has different resource definitions then what is actually used for the step that runs is undefined?

Hmm, yeah, that is problematic. Do you think having the limits as parts of the cache key would be better? I think it's common to set the same limits for many builds, which would still work correctly and prevent "undefined" behavior, which is present in the current implementation.

tonistiigi · 2026-03-10T22:51:33Z

No, having them in cache key would make the steps run multiple times, and cache import to not work if resource limits do not match, what I think is worse.

Ideally, we can make it so that the more relaxed resource limits wins. Or even that the build that was started before wins would be improvement. These cases may be simpler to implement if the resources are not part of the LLB checksum(eg. can't be inside Meta then). If this becomes too complicated, just documenting that behavior is undefined may be an option.

jirimoravcik · 2026-03-12T18:28:37Z

No, having them in cache key would make the steps run multiple times, and cache import to not work if resource limits do not match, what I think is worse.

Ideally, we can make it so that the more relaxed resource limits wins. Or even that the build that was started before wins would be improvement. These cases may be simpler to implement if the resources are not part of the LLB checksum(eg. can't be inside Meta then). If this becomes too complicated, just documenting that behavior is undefined may be an option.

Ok, makes sense. I implemented logic that chooses the most "relaxed" resources. Happy to address any feedback. Thanks!

jirimoravcik · 2026-03-20T08:06:16Z

@tonistiigi Hello, rebased the PR to have one commit only. Can you run the CI? Thanks

islewis · 2026-04-20T21:55:44Z

any update on this? this feature would be incredibly helpful

jirimoravcik · 2026-05-05T08:20:00Z

any update on this? this feature would be incredibly helpful

I'll take a look this week

tonistiigi

Can't we do it similarly how the cache sources are handled in
https://github.com/moby/buildkit/blob/v0.30.0/solver/jobs.go#L618-L624

So if vertex exists then the resources in metadata are merged at load time and kept in st. You should be able to read it out in ResolveOp and pass directly to NewExecOp there.

I told you I don't want LLB proto dependencies in solver pkg so you can't add smth like v.Options().PBResources directly in the vertex type, but we could add something that is defined by interface that has a Merge(another) function. You can get back the real proto based type in ResolveOp`.

tonistiigi · 2026-05-20T02:15:23Z

+		if err != nil {
+			return nil, err
+		}
+		if eo, ok := op.(*ops.ExecOp); ok {


This should be inside the ResolveOp implementation. Pass it to NewExecOp constructor if you need to.

I changed the implementation, it's now passed to NewExecOp constructor. Do you think there's an alternative way to do it?

tonistiigi · 2026-05-20T02:17:56Z

 		Ulimit:                    e.op.Meta.Ulimit,
 		CDIDevices:                e.op.CdiDevices,
 		CgroupParent:              e.op.Meta.CgroupParent,
+		LinuxResources:            MergeLinuxResourcesForVertex(e.builder, e.digest),


I would prefer if this merge happens on LLB load, not via some callback during ops execution. The current one breaks the worker-solver separation as it just calls back to solver during worker ops run. The resources should be determined and then passed to NewExecOp directly for example.

I'm currently storing "generic" metadata on VertexOptions and extracting the linux resources from that afterwards.

The merge now happens inside loadUnlocked

jirimoravcik · 2026-05-20T21:37:59Z

Can't we do it similarly how the cache sources are handled in https://github.com/moby/buildkit/blob/v0.30.0/solver/jobs.go#L618-L624

So if vertex exists then the resources in metadata are merged at load time and kept in st. You should be able to read it out in ResolveOp and pass directly to NewExecOp there.

I told you I don't want LLB proto dependencies in solver pkg so you can't add smth like v.Options().PBResources directly in the vertex type, but we could add something that is defined by interface that has a Merge(another) function. You can get back the real proto based type in ResolveOp`.

Hey, thanks for the feedback. I reworked some parts of the PR, it sadly grew from the original (which I though would be simple), sorry for that.

tonistiigi

I quite like the latest variant. Some small comments.

tonistiigi · 2026-05-26T22:38:13Z

+		if err1 != nil || err2 != nil || loN < 0 || hiN < loN {
+			return nil, errors.Errorf("invalid cpuset range %q", part)
+		}
+		for i := loN; i <= hiN; i++ {


This should have a fixed max cap (eg. 4096?). Atm you can pass a very big number and then daemon will allocate a map with that map with that many items.

Good point. Fixed. I set the value to 8192, which is used as the normal maximum of CPUs in moby, see https://github.com/moby/moby/blob/ad6b80a888d3248fba66ce149f1f072487f7697b/pkg/sysinfo/sysinfo_linux.go#L337

Add support for setting cgroup resource limits (memory, memory-swap, cpu-shares, cpu-period, cpu-quota, cpuset-cpus, cpuset-mems) on individual build steps. Signed-off-by: Jiří Moravčík <jiri.moravcik@gmail.com>

tonistiigi · 2026-05-28T22:35:24Z

@jirimoravcik Can you follow this up with Buildx changes as well.

jirimoravcik · 2026-05-29T10:38:58Z

@jirimoravcik Can you follow this up with Buildx changes as well.

docker/buildx#3876

Port of moby/buildkit#6569 to buildx. Adds --memory, --memory-swap, --cpu-shares, --cpu-period, --cpu-quota, --cpuset-cpus, and --cpuset-mems flags to build, plus the equivalent bake target attributes and compose x-bake fields. Signed-off-by: Jiří Moravčík <jiri.moravcik@gmail.com>

Copilot AI review requested due to automatic review settings March 10, 2026 15:50

github-actions Bot added area/dockerfile area/llb area/testing area/windows area/client labels Mar 10, 2026

github-actions Bot assigned jirimoravcik Mar 10, 2026

github-actions Bot added area/frontend area/executor area/solver labels Mar 10, 2026

Copilot started reviewing on behalf of jirimoravcik March 10, 2026 15:51 View session

Copilot AI reviewed Mar 10, 2026

View reviewed changes

Comment thread client/llb/state.go

Comment thread executor/oci/spec_linux.go Outdated

github-actions Bot added area/docs area/source labels Mar 12, 2026

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch from c48b5d8 to 7546f2e Compare March 20, 2026 08:04

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch from 7546f2e to e24c17a Compare March 29, 2026 11:47

tonistiigi reviewed Apr 21, 2026

View reviewed changes

Comment thread solver/jobs.go Outdated

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch from e24c17a to a75be90 Compare May 16, 2026 16:52

github-actions Bot removed the area/source label May 16, 2026

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch from a75be90 to d91bd1b Compare May 16, 2026 17:17

jirimoravcik requested a review from tonistiigi May 16, 2026 17:43

tonistiigi reviewed May 20, 2026

View reviewed changes

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch from d91bd1b to a15f6be Compare May 20, 2026 09:52

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch 2 times, most recently from 81ccc1e to f44893b Compare May 20, 2026 20:38

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch 2 times, most recently from 3cec68a to 61f0ee7 Compare May 20, 2026 21:51

jirimoravcik requested a review from tonistiigi May 20, 2026 21:52

tonistiigi reviewed May 21, 2026

View reviewed changes

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch from 61f0ee7 to 5cc8359 Compare May 25, 2026 12:52

github-actions Bot added the area/util label May 25, 2026

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch from 5cc8359 to 683c37c Compare May 25, 2026 13:00

jirimoravcik requested a review from tonistiigi May 25, 2026 13:16

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch 3 times, most recently from 658f1ef to 69a6c86 Compare May 25, 2026 13:59

tonistiigi reviewed May 26, 2026

View reviewed changes

solver: add per-step CPU and memory resource limits

f85c740

Add support for setting cgroup resource limits (memory, memory-swap, cpu-shares, cpu-period, cpu-quota, cpuset-cpus, cpuset-mems) on individual build steps. Signed-off-by: Jiří Moravčík <jiri.moravcik@gmail.com>

jirimoravcik force-pushed the feat/add-cpu-memory-limits branch from 69a6c86 to f85c740 Compare May 27, 2026 09:34

jirimoravcik requested a review from tonistiigi May 27, 2026 09:39

tonistiigi added this to the v0.31.0 milestone May 27, 2026

tonistiigi approved these changes May 27, 2026

View reviewed changes

tonistiigi merged commit f5ec43c into moby:master May 27, 2026
192 checks passed

jirimoravcik mentioned this pull request May 29, 2026

build: Add per-step CPU and memory resource limits docker/buildx#3876

Merged

Conversation

jirimoravcik commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

tonistiigi commented Mar 10, 2026

Uh oh!

jirimoravcik commented Mar 10, 2026

Uh oh!

tonistiigi commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jirimoravcik commented Mar 12, 2026

Uh oh!

jirimoravcik commented Mar 20, 2026

Uh oh!

islewis commented Apr 20, 2026

Uh oh!

Uh oh!

jirimoravcik commented May 5, 2026

Uh oh!

tonistiigi left a comment

Choose a reason for hiding this comment

Uh oh!

tonistiigi May 20, 2026

Choose a reason for hiding this comment

Uh oh!

jirimoravcik May 20, 2026

Choose a reason for hiding this comment

Uh oh!

tonistiigi May 20, 2026

Choose a reason for hiding this comment

Uh oh!

jirimoravcik May 20, 2026

Choose a reason for hiding this comment

Uh oh!

jirimoravcik commented May 20, 2026

Uh oh!

tonistiigi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tonistiigi May 26, 2026

Choose a reason for hiding this comment

Uh oh!

jirimoravcik May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tonistiigi commented May 28, 2026

Uh oh!

jirimoravcik commented May 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jirimoravcik commented Mar 10, 2026 •

edited

Loading

tonistiigi commented Mar 10, 2026 •

edited

Loading

jirimoravcik May 27, 2026 •

edited

Loading