feat: on arm64 (Grace-Blackwell GB200/GB300) skip fabric manager and install nvidia-imex

[Xu Xue (xuexu6666)]

On arm64 Grace-Blackwell (GB200/GB300), cross-node NVLink is coordinated by IMEX, not a node-local fabric manager. This PR makes the aks-gpu image do the right thing on GB, in the same code path where x86 installs fabric manager:

1. Skip fabric manager on arm64

The FM installer looked for fabricmanager-linux-aarch64-… (from uname -m), which doesn't exist (the redist archive is sbsa) → exit 127 → CSE retries the whole container → ~780s timeout → exit 84 → ensureKubelet never runs → the GB node never joins the cluster. GB has no node-local fabric manager, so FM is now skipped on arm64 (download + install).

2. Install nvidia-imex on arm64

nvidia-imex (the cross-node NVLink / ComputeDomains coordinator) is not in the driver .run or the fabric-manager redist — it ships only as a separate deb in the CUDA repo. So:

• download.sh bundles the version-matched nvidia-imex arm64 deb into the image, resolving the exact filename from the repo Packages index (the deb revision suffix, e.g. -1ubuntu1, varies by version), OS path adapted via VERSION_ID.
• install.sh installs it at node boot via dpkg -i --force-depends (the deb declares nvidia-modprobe, which the runfile driver provides on-disk but not as a deb) and leaves the node-wide nvidia-imex.service OFF — IMEX is orchestrated per-ComputeDomain by the NVIDIA DRA driver, which CDI-injects the host binary into its daemon pods.

Validation

Proven end-to-end on a managed-driver GB300 cluster: with nvidia-imex installed, the DRA ComputeDomain reaches Ready and cross-node MNNVL NCCL all_reduce runs via P2P/MNNVL at ~698 GB/s (peak). x86 behavior is unchanged.

[Xu Xue (xuexu6666)]

Code review

No issues found. Checked for bugs and shell-scripting correctness (repo has no CLAUDE.md).

Reviewed the arm64 (GB200/GB300) fabric-manager skip + nvidia-imex install across download.sh and install.sh from five angles (bug scan, git history, prior-PR feedback, code-comment compliance, conventions). The design holds up:

• Arch gates are correct and symmetric: download.sh uses build-arg TARGETARCH == arm64, install.sh uses runtime ARCH == aarch64 (uname -m); x86/GRID paths are unchanged.
• install_imex version-pins to nvidia-imex_${DRIVER_VERSION}-, resolves exactly one deb from the sbsa Packages index, and exit 1s if none is found.
• The awk first-match flag (no early exit) correctly avoids SIGPIPE-ing curl under set -o pipefail, matching its comment.
• dpkg -i --force-depends (runfile driver satisfies nvidia-modprobe on disk) and systemctl disable --now nvidia-imex.service || true (left off for DRA per-ComputeDomain orchestration) both match their comments.

Non-blocking observations, not posted as defects: the dpkg -i /opt/gpu/nvidia-imex_*_arm64.deb glob is unpinned, but /opt/gpu is a fresh build layer with exactly one staged deb, so there is no realistic multi/zero-match path (and it matches the existing install_cached_nvidia_packages convention). Since the PR is branched behind main, the lint-and-test (shellcheck+bats) gate from #166 does not run here and there is no bats coverage for the new arm64 branch — a rebase would pick that up.