Attribute DCGM exporter metrics to pods on DRA nodes by default

[rajathagasthya]

Description

Under GPUCluster, dcgm-exporter metrics carry only hardware labels (no pod, namespace, container or dra_*) unless dcgmExporter.enablePodLabels is set. ClusterPolicy has those labels by default, so migrating to GPUCluster silently loses attribution.

The kubelet reports device-plugin allocations as GPU UUIDs, which the exporter gets from the pod-resources socket with no API access. DRA allocations are reported as (pool, device) names instead, so the exporter has to read the node's ResourceSlice to resolve them back to a UUID. That needs a ServiceAccount token and read access to resourceslices — both of which, along with KUBERNETES_ENABLE_DRA itself, were gated behind the pod-labels flag.

Render the DRA env var, token mount and ClusterRole/binding unconditionally. enablePodLabels and enablePodUID controls their behavior appropriately.

Builds on #2697, which added the env var and the resourceslices grant, but gated them behind PodMetadataEnabled gate. PodMetadataEnabled now has no remaining consumers and is removed.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

Verified on a GH200 node, driver 595.71.05, with a single DRA workload holding one GPU.

With PR changes

Default pod attribution (pod label and pod UID false) — matches current ClusterPolicy behavior

$ curl -s localhost:9400/metrics | grep 'pod="gpu-workload-dra-859cdbfb47-47dll"' | head -1
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS{gpu="0",UUID="GPU-cbd42015-907e-1554-4c37-4490ecc6216e",pci_bus_id="00000009:01:00.0",device="nvidia0",modelName="NVIDIA GH200 480GB",hostname="<node>",DCGM_FI_DRIVER_VERSION="595.71.05",container="trainer",dra_claim_name="gpu-workload-dra-859cdbfb47-47dll-shared-gpu-974nq",dra_claim_namespace="gpu-operator",dra_device_name="gpu-0",dra_driver_name="gpu.nvidia.com",dra_pool_name="<node>",namespace="gpu-operator",pod="gpu-workload-dra-859cdbfb47-47dll"} 0

No pod label leakage

$ curl -s localhost:9400/metrics | grep 'pod="gpu-workload-dra-859cdbfb47-47dll"' | grep -c 'attribution-label'
0

With pod label and UID enabled (no pod UID?)

$ curl -s localhost:9400/metrics | grep 'pod="gpu-workload-dra-859cdbfb47-47dll"' | head -1
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS{gpu="0",UUID="GPU-cbd42015-907e-1554-4c37-4490ecc6216e",pci_bus_id="00000009:01:00.0",device="nvidia0",modelName="NVIDIA GH200 480GB",hostname="<node>",DCGM_FI_DRIVER_VERSION="595.71.05",app="gpu-workload-dra",dra="attribution-label",pod_template_hash="859cdbfb47",container="trainer",dra_claim_name="gpu-workload-dra-859cdbfb47-47dll-shared-gpu-974nq",dra_claim_namespace="gpu-operator",dra_device_name="gpu-0",dra_driver_name="gpu.nvidia.com",dra_pool_name="<node>",namespace="gpu-operator",pod="gpu-workload-dra-859cdbfb47-47dll"} 0

$ curl -s localhost:9400/metrics | grep 'pod="gpu-workload-dra-859cdbfb47-47dll"' | grep -c 'attribution-label'
23

The absent pod UID is not caused by this change. dcgm-exporter never sets the UID attribute on its DRA path, so DCGM_EXPORTER_KUBERNETES_ENABLE_POD_UID has no effect on DRA-attributed metrics. The ClusterPolicy run below shows pod_uid present on the same exporter build, which confirms the gap in DRA path.

Without PR changes

No match (pod labels and pod UID false) - current bug

v26.7.0-rc.4, which predates #2697 and never sets KUBERNETES_ENABLE_DRA so no pods are matched.

$ curl -s localhost:9400/metrics | grep 'pod="gpu-workload-dra-859cdbfb47-9g7gs"' | head -1

All metrics

# HELP DCGM_FI_PROF_PCIE_RX_BYTES The rate of data received over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
# TYPE DCGM_FI_PROF_PCIE_RX_BYTES gauge
DCGM_FI_PROF_PCIE_RX_BYTES{gpu="0",UUID="GPU-cbd42015-907e-1554-4c37-4490ecc6216e",pci_bus_id="00000009:01:00.0",device="nvidia0",modelName="NVIDIA GH200 480GB",hostname="<node>",DCGM_FI_DRIVER_VERSION="595.71.05"} 167781
# HELP DCGM_FI_PROF_PCIE_TX_BYTES The rate of data transmitted over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
# TYPE DCGM_FI_PROF_PCIE_TX_BYTES gauge
DCGM_FI_PROF_PCIE_TX_BYTES{gpu="0",UUID="GPU-cbd42015-907e-1554-4c37-4490ecc6216e",pci_bus_id="00000009:01:00.0",device="nvidia0",modelName="NVIDIA GH200 480GB",hostname="<node>",DCGM_FI_DRIVER_VERSION="595.71.05"} 152781
# HELP DCGM_FI_PROF_PIPE_TENSOR_ACTIVE Ratio of cycles the tensor (HMMA) pipe is active.
# TYPE DCGM_FI_PROF_PIPE_TENSOR_ACTIVE gauge
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{gpu="0",UUID="GPU-cbd42015-907e-1554-4c37-4490ecc6216e",pci_bus_id="00000009:01:00.0",device="nvidia0",modelName="NVIDIA GH200 480GB",hostname="<node>",DCGM_FI_DRIVER_VERSION="595.71.05"} 0.853072

Pod labels (and pod UID) enabled — with PR #2697

Attribution works once enablePodLabels=true is set, confirming the gate is the only thing this PR changes.

$ curl -s localhost:9400/metrics | grep 'pod="gpu-workload-dra-859cdbfb47-qw5jk"' | head -1
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS{gpu="0",UUID="GPU-cbd42015-907e-1554-4c37-4490ecc6216e",pci_bus_id="00000009:01:00.0",device="nvidia0",modelName="NVIDIA GH200 480GB",hostname="<node>",DCGM_FI_DRIVER_VERSION="595.71.05",app="gpu-workload-dra",dra="attribution-label",pod_template_hash="859cdbfb47",container="trainer",dra_claim_name="gpu-workload-dra-859cdbfb47-qw5jk-shared-gpu-rcd7q",dra_claim_namespace="gpu-operator",dra_device_name="gpu-0",dra_driver_name="gpu.nvidia.com",dra_pool_name="<node>",namespace="gpu-operator",pod="gpu-workload-dra-859cdbfb47-qw5jk"} 0

$ curl -s localhost:9400/metrics | grep 'pod="gpu-workload-dra-859cdbfb47-qw5jk"' | grep -c 'attribution-label'
23

ClusterPolicy

No regression.

Default attribution

$ curl -s localhost:9400/metrics | grep 'pod="gpu-workload-5dd46f4645-2nmm7"' | head -1
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS{gpu="0",UUID="GPU-cbd42015-907e-1554-4c37-4490ecc6216e",pci_bus_id="00000009:01:00.0",device="nvidia0",modelName="NVIDIA GH200 480GB",hostname="<node>",DCGM_FI_DRIVER_VERSION="595.71.05",container="trainer",namespace="gpu-operator",pod="gpu-workload-5dd46f4645-2nmm7"} 0

Pod labels and pod uid enabled (app=gpu-workload)

$ curl -s localhost:9400/metrics | grep 'pod="gpu-workload-5dd46f4645-2nmm7"' | head -1
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS{gpu="0",UUID="GPU-cbd42015-907e-1554-4c37-4490ecc6216e",pci_bus_id="00000009:01:00.0",device="nvidia0",modelName="NVIDIA GH200 480GB",hostname="<node>",DCGM_FI_DRIVER_VERSION="595.71.05",app="gpu-workload",pod_template_hash="5dd46f4645",container="trainer",namespace="gpu-operator",pod="gpu-workload-5dd46f4645-2nmm7",pod_uid="190f53e6-7b28-4a97-b1e6-112c2dea6b47"} 0

$ curl -s localhost:9400/metrics | grep 'pod="gpu-workload-5dd46f4645-2nmm7"' | grep -c "app="
23

[rajathagasthya]

@karthikvetrivel From my testing:

The absent pod UID is not caused by this change. dcgm-exporter never sets the UID attribute on its DRA path, so DCGM_EXPORTER_KUBERNETES_ENABLE_POD_UID has no effect on DRA-attributed metrics. The ClusterPolicy run below shows pod_uid present on the same exporter build, which confirms the gap in DRA path.

Not sure if you're already aware of this issue. This needs to be reported to upstream DCGM Exporter.