[Bug]: nvidia-vgpu-device-manager DaemonSet has no priorityClassName, unlike every other GPU Operator DaemonSet — exposed to permanently-stuck-Pending pattern during node remediation

[yogeshbendre]

---

Describe the bug

How this was found: by auditing every DaemonSet-bearing component the GPU Operator ships (checked at tag v26.3.3, commit b0a49c0e7b2e061dcd83f2bb2fe4fe960c5d0338) for priorityClassName, then confirming the result by deploying the actual, unmodified manifest to a live cluster — not a static-analysis-only claim.

Of the 13 DaemonSet templates under assets/, 12 hardcode priorityClassName: system-node-critical:

• state-driver, state-device-plugin, state-container-toolkit, gpu-feature-discovery, state-dcgm-exporter, state-mig-manager, state-vgpu-manager, state-cc-manager, state-kata-device-plugin, state-kata-manager, state-mps-control-daemon, state-node-status-exporter, state-operator-validation, state-sandbox-device-plugin, state-sandbox-validation, state-vfio-manager

nvidia-vgpu-device-manager is the sole exception — assets/state-vgpu-device-manager/0600_daemonset.yaml has no priorityClassName field anywhere.

This isn't compensated for elsewhere: controllers/object_controls.go#L819-L822 (applyCommonDaemonsetConfig, applied to every DaemonSet) only overrides PriorityClassName when an operator explicitly sets a cluster-wide ClusterPolicy.spec.daemonsets.priorityClassName — it never injects a default for a DaemonSet whose own template omits the field:

// update PriorityClass
if config.Daemonsets.PriorityClassName != "" {
    obj.Spec.Template.Spec.PriorityClassName = config.Daemonsets.PriorityClassName
}

Confirmed live, deploying the two real manifests (namespace/image placeholders substituted, no other changes) to a kind cluster:

$ kubectl get pods -o jsonpath='{.spec.priority}{" "}{.spec.priorityClassName}{"\n"}'
nvidia-device-plugin-daemonset-...   2000001000 system-node-critical
nvidia-vgpu-device-manager-...       0

nvidia-vgpu-device-manager admits at priority 0 — identical to any ordinary, unprivileged workload pod.

Why this matters: kube-scheduler's default preemption plugin only allows a pod to preempt a strictly lower-priority victim (isPreemptionAllowed, a strict <, not <=). This is the exact mechanism behind kubernetes/kubernetes#140984: a node-pinned pod at default priority can be permanently locked out if its one eligible node fills up with other equal-priority pods first, with no self-heal, because preemption can never reclaim the slot. nvidia-vgpu-device-manager is a DaemonSet — hard-pinned to whichever node it targets — running at that same default priority. During a mass node remediation event (health-triggered cordon/drain/reboot across a fleet), if the node comes back and fills with equal-priority workload pods before this DaemonSet's replacement pod is considered, it can be stuck Pending indefinitely. Since this DaemonSet applies vGPU device/profile configuration on the node, that node's vGPU-sliced GPU capacity may never get (re)configured after the remediation event — with nothing to signal that a stuck DaemonSet pod, rather than a hardware issue, is the cause.

---

To Reproduce

No real GPU hardware or vGPU license is required — this is a manifest/admission-level check, not a functional GPU test.

# 1. Any Kubernetes cluster (a plain `kind` cluster is sufficient)
kind create cluster --name vgpu-dm-repro

# 2. Fetch the actual, unmodified DaemonSet manifest from the operator's assets at the pinned tag
curl -sSL -o vgpu-dm.yaml \
  https://raw.githubusercontent.com/NVIDIA/gpu-operator/v26.3.3/assets/state-vgpu-device-manager/0600_daemonset.yaml

# 3. Substitute the operator-templated placeholders (namespace, image) with real values —
#    this is the ONLY change needed; priorityClassName is absent in the source file itself
sed -i 's#"FILLED BY THE OPERATOR"#default#; s#image: default#image: "registry.k8s.io/pause:3.10"#g' vgpu-dm.yaml

# 4. Provide the ServiceAccount the manifest references, and label a node to match its nodeSelector
kubectl create serviceaccount nvidia-vgpu-device-manager
kubectl label node <a-node> nvidia.com/gpu.deploy.vgpu-device-manager=true

# 5. Apply and check the admitted priority
kubectl apply -f vgpu-dm.yaml
kubectl get pods -l app=nvidia-vgpu-device-manager \
  -o jsonpath='{.items[0].spec.priority}{" "}{.items[0].spec.priorityClassName}{"\n"}'
# -> 0   (empty priorityClassName)

For contrast, repeating the same steps with assets/state-device-plugin/0500_daemonset.yaml (which does set priorityClassName: system-node-critical in its source) shows 2000001000 system-node-critical instead.

---

Expected behavior

nvidia-vgpu-device-manager should run at priorityClassName: system-node-critical, consistent with every other DaemonSet the GPU Operator ships, so it has the same protection against being permanently starved of scheduling by equal-priority workload pods during node remediation events.

---

Environment (please provide the following information):

• GPU Operator Version: v26.3.3 (commit b0a49c0e7b2e061dcd83f2bb2fe4fe960c5d0338) — root cause is a static manifest omission, not version-specific behavior; not yet checked against main/unreleased commits for whether it has already been fixed.
• Reproduction environment: vanilla kind cluster, no cloud provider, no real GPU/vGPU hardware — the bug is at the pod-admission/priority level, independent of the driver or vGPU functionality itself.

---

Information to attach (optional if deemed irrelevant)

Not applicable — this is a manifest-completeness finding confirmed via kubectl get pod -o jsonpath on the priority/priorityClassName fields, not a runtime error requiring pod logs or a nvidia-smi capture. Happy to provide the full kubectl describe/get -o yaml output for both DaemonSets used in the reproduction if useful for triage.

[rajathagasthya]

Thanks for the writeup, but this is not a bug.

The chart sets a default priority class that the operator applies to all daemonsets, so the vgpu-device-manager lands with system-node-critical already.

gpu-operator/deployments/gpu-operator/values.yaml

Lines 41 to 44 in 14ca11b

	daemonsets:
	labels: {}
	annotations: {}
	priorityClassName: system-node-critical

Probably worth adding it to the manifest for consistency, but since it has no functional impact I'm going to close this.

[rajathagasthya]

On second thought, fixing this in the manifests probably adds an additional safety layer on OpenShift clusters since it doesn't use helm defaults. Although, the problem is overstated in the description — equal-priority workload pods won't block this daemonset indefinitely as this does not make resource requests.

[yogeshbendre]

Appreciate you digging into this, @rajathagasthya — both points are fair, and you're right that I tested by deploying the raw state manifest directly rather than through actual operator reconciliation, which is why I missed the chart-default injection.
Glad it's still worth the manifest fix for non-Helm-default installs.

[yogeshbendre]

Fair's fair — you got a second thought, I'll take one too :)

On reflection, the point about "won't be blocked indefinitely because it has no resource requests" doesn't seem entirely true.

Kubernetes' NodeResourcesFit plugin checks a node's pod-count ceiling (status.allocatable.pods, kubelet's --max-pods, default 110) for every pod unconditionally — this happens before any CPU/memory check, so it applies regardless of what the pod itself requests:

// pkg/scheduler/framework/plugins/noderesources/fit.go
allowedPodNumber := nodeInfo.GetAllocatable().GetAllowedPodNumber()
if len(nodeInfo.GetPods())+1 > allowedPodNumber {
insufficientResources = append(insufficientResources, InsufficientResource{
ResourceName: v1.ResourcePods,
Reason: "Too many pods",

So a zero-request pod like nvidia-vgpu-device-manager is exposed to the same equal-priority preemption gap once a node's pod-count fills up — no CPU/memory pressure needed, just pod density.

Doesn't change the reconciliation-methodology point (that one's on me), but the conclusion of "no functional impact" may not do justice to the full picture :)

Any thoughts?