Skip to content

Remove mode label and do not allow ClusterPolicy & GPUCluster co-existence - #2696

Merged
cdesiniotis merged 1 commit into
NVIDIA:mainfrom
cdesiniotis:remove-mode-label
Jul 31, 2026
Merged

Remove mode label and do not allow ClusterPolicy & GPUCluster co-existence#2696
cdesiniotis merged 1 commit into
NVIDIA:mainfrom
cdesiniotis:remove-mode-label

Conversation

@cdesiniotis

@cdesiniotis cdesiniotis commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Description

This commit removes the 'nvidia.com/gpu-operator.resource-allocation.mode' label. The node-labeling-controller no longer adds this label to nodes or uses it when determining what software stack to deploy on a node. In addition, this commit makes it so a ClusterPolicy CR and GPUCluster CR cannot co-exist in the same cluster. Only one software stack (device-plugin or dra) will be deployed in the cluster.

In the future, we may remove this limitation and allow both stacks to co-exist in the same cluster once we decide on a mechanism for users to choose which stack to deploy per-node.

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint)
  • Generated assets in-sync (make validate-generated-assets)
  • Go mod artifacts in-sync (make validate-modules)
  • Test cases are added for new code paths

Testing

Manually tested on a single GPU node. Tested several scenarios:

  • Only a ClusterPolicy CR deployed (default helm values)
  • Only a GPUCluster CR deployed (--set clusterPolicy.deployCR=false --set gpuCluster.deployCR=true)
  • Both ClusterPolicy and GPUCluster CRs deployed. Confirmed that the status of both CRs is reflected correctly and GPU Operator logs contain relevant errors.

@cdesiniotis
cdesiniotis force-pushed the remove-mode-label branch 8 times, most recently from 3e55040 to 8fb2f3f Compare July 30, 2026 20:49
@cdesiniotis cdesiniotis self-assigned this Jul 30, 2026
@cdesiniotis
cdesiniotis marked this pull request as ready for review July 30, 2026 21:10
# clusterPolicy controls whether the chart deploys the ClusterPolicy CR, which
# manages the classic device-plugin GPU enablement stack. Disable it to run a
# GPUCluster-only (DRA) installation.
# The ClusterPolicy CR stores the desired state of the GPU

@cdesiniotis cdesiniotis Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The updates to the helm values.yaml file here are the only "user-facing" change in this PR. Let's review this carefully.

Comment thread deployments/gpu-operator/values.yaml
Comment thread controllers/clusterpolicy_controller.go Outdated

@rahulait rahulait left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, LGTM

Comment thread controllers/clusterpolicy_controller.go
Comment thread manifests/state-dcgm-exporter/0700_daemonset.yaml

@rajathagasthya rajathagasthya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Comment on lines +5 to +7
{{- if and .Values.gpuCluster.deployCR (and .Values.driver.enabled (not .Values.driver.nvidiaDriverCRD.enabled)) }}
{{ fail "the NVIDIADriver CRD must be enabled when deploying a GPUCluster CR, set driver.nvidiaDriverCRD.enabled=true" }}
{{- end }}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this helm validation to ensure the NVIDIADriver CRD is enabled when users try to deploy a GPUCluster CR.

…tence

This commit removes the 'nvidia.com/gpu-operator.resource-allocation.mode' label.
The node-labeling-controller no longer adds this label to nodes or uses it when
determining what software stack to deploy on a node. In addition, this commit
makes it so a ClusterPolicy CR and GPUCluster CR cannot co-exist in the same
cluster. Only one software stack (device-plugin or dra) will be deployed in
the cluster.

Signed-off-by: Christopher Desiniotis <cdesiniotis@nvidia.com>
@cdesiniotis
cdesiniotis merged commit 2afe99f into NVIDIA:main Jul 31, 2026
19 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants