fix(e2e): replicate gallery images to a fixed region set

[Ganeshkumar Ashokavardhanan (ganeshkumarashok)]

What this PR does / why we need it:

GPU E2E intermittently fails with GalleryImageNotFound. This fixes the race that causes it, and replaces the region bookkeeping with a fixed, generated list.

Root cause

A gallery image version's PublishingProfile.TargetRegions is full desired state — an update replaces the region list rather than appending to it.

ensureReplication read the live version, appended only the caller's own region, and wrote it back. CachedPrepareVHD is keyed on {Image, Location} (e2e/cache.go), so the same image prepared for two different regions is not deduped and two goroutines run that read-modify-write concurrently:

both read TargetRegions = R0
writer A (westus2)  -> writes R0 ∪ {westus2}
writer B (uaenorth) -> writes R0 ∪ {uaenorth}   # drops westus2

The test whose region was dropped then fails with GalleryImageNotFound at VMSS create.

The fix

Make the desired region set independent of the caller. Every writer then submits an identical desired state, so interleaved reads and writes converge — a lost update loses nothing, and no locking, merging or conflict resolution is needed.

The set is hardcoded in exactly one place, e2e/config/regions.go. To stop it drifting from the scenarios, the per-OS lists in regions_generated.go are derived from the scenario files by e2e/hack/genregions:

make generate-e2e-regions   # after pinning a scenario to a new region
make validate-e2e-regions   # staleness gate (also enforced by a unit test)

Scenarios now reference config.Region* constants instead of raw literals, and the generator errors on any Location it cannot resolve statically — including literals passed through a helper's location parameter — so a new region cannot silently miss the replication set.

Current sets: Linux eastus, southcentralus, southeastasia, uaenorth, westus2, westus3; Windows westus3 only (no Windows scenario pins a region, and those images are large enough that replicating them where no Windows test looks is pure cost).

Also in this PR

• An unsupported non-ephemeral region is now an error rather than being appended at runtime. Appending is what made desired state caller-dependent in the first place; widening the set is a code change, not a runtime decision. This also closes the cross-process variant of the bug, where two pipelines with different E2E_LOCATION would clobber each other.
• Concurrent writers can still collide on the very first replication, so a 409/412 is retried with backoff. The retry only re-reads and re-checks — it never reconciles, because there is nothing to reconcile.
• maybeSkipScenario fails fast with the actual fix instead of a confusing GalleryImageNotFound much later.
• Region comparisons go through NormalizeRegion, so "West US 2" and "westus2" match. (strings.EqualFold did not.)

Deliberately not included: replication-status polling or target-region merging. Those exist to reconcile divergent desired states; once desired state is constant there is nothing to reconcile. CreateVMSSWithRetry already retries GalleryImageNotFound, which is the right layer for genuine fabric-level eventual consistency.

Testing

• 10 unit tests in e2e/config, 4 in e2e/hack/genregions.
• Verified by hand that the drift gate fires on a stale generated file, and that the generator rejects a region literal both in a Scenario.Location field and passed through a helper argument.
• go build ./..., go vet ./..., go test ./config/... ./hack/..., make validate-e2e-regions all clean.

Which issue(s) this PR fixes:

N/A

[github-actions]

Windows Unit Test Results

  3 files   12 suites   47s ⏱️
389 tests 389 ✅ 0 💤 0 ❌
392 runs  392 ✅ 0 💤 0 ❌

Results for commit 9177885.

♻️ This comment has been updated with latest results.

[Copilot]

Pull request overview

Fixes concurrent gallery image replication races by using deterministic, generated region sets.

Changes:

• Adds fixed Linux and Windows replication regions with conflict retries.
• Generates region lists from E2E scenarios and validates drift.
• Replaces pinned region literals with shared constants.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
Makefile	Adds region generation and validation targets.
e2e/test_helpers.go	Validates region support and marks temporary images ephemeral.
e2e/scenario_test.go	Uses region constants.
e2e/scenario_gpu_managed_experience_test.go	Uses region constants.
e2e/hack/genregions/main.go	Implements scenario-region generation.
e2e/hack/genregions/main_test.go	Tests generator behavior and drift.
e2e/config/vhd.go	Adds ephemeral-image metadata and normalized comparisons.
e2e/config/regions.go	Defines region constants and replication policies.
e2e/config/regions_test.go	Tests region and conflict behavior.
e2e/config/regions_generated.go	Contains generated replication sets.
e2e/config/azure.go	Applies fixed replication sets with retries.

Files not reviewed (1)

• e2e/config/regions_generated.go: Generated file

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated no new comments.

Files not reviewed (1)

• e2e/config/regions_generated.go: Generated file

Suppressed comments (1)

e2e/hack/genregions/main.go:96

• 🟡 Medium Risk — 🔧 Script Logic: This glob excludes helpers defined in non-test files, so locationParameters never sees the existing runScenarioUbuntu2404GPUNPD in e2e/test_helpers.go:1045. A future call such as runScenarioUbuntu2404GPUNPD(t, sku, "newregion", "") would therefore pass this drift gate and omit the region, recreating GalleryImageNotFound. Parse helper declarations from all root Go files while deriving scenario usage from test files, and cover a helper/test split across separate files.

	files, err := filepath.Glob(filepath.Join(root, "*_test.go"))

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (1)

e2e/config/regions_test.go:49

• 🟡 Medium Risk — 🧪 Test Coverage: This regex only catches literals assigned directly to Location; it cannot catch a literal passed through helpers such as runScenarioACLGPU(..., location) (scenario_test.go:281-284), and it does not derive separate Linux/Windows region sets. A call like runScenarioACLGPU(t, ..., "northeurope") therefore passes this CI gate and fails only at runtime in maybeSkipScenario. The PR description promises an AST generator, generated per-OS lists, and a staleness command, but none of those files/targets are present in this change. Please add and run that generator, or otherwise make the gate resolve helper arguments and validate the per-OS sets.

	literal := regexp.MustCompile(`(?m)^\s*Location:\s*"([^"]*)"`)

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (4)

e2e/config/regions_test.go:20

• 🟡 Medium Risk — 🧪 Test Coverage: This deliberately leaves helper-propagated locations outside the CI gate, even though existing scenarios use exactly that pattern (for example scenario_test.go:266-284 and scenario_test.go:2994-2998). Because this workflow only runs go test ./config/..., a new helper argument can be merged without entering the fixed replication set and will fail only when that E2E scenario runs. Please implement the described static generator/staleness validation (including helper arguments) and execute it in CI rather than relying on this regex scan.

// A scenario naming a region that images are not replicated to would fail at runtime, so the
// inline regions are checked against e2eRegions here instead. Regions passed to a helper
// rather than set on the struct are not visible to this scan; maybeSkipScenario catches those

e2e/config/regions_test.go:38

• 🟡 Medium Risk — 🧪 Test Coverage: This validates every literal against the Linux-wide e2eRegions, but replicationRegions permits Windows images only in westus3. A future Windows scenario pinned to an existing Linux region such as westus2 therefore passes this CI check and then fails in maybeSkipScenario. Validate against generated per-OS region sets (or otherwise resolve each scenario's image OS) so the drift gate matches runtime behavior.

			if !slices.Contains(e2eRegions, NormalizeRegion(match[1])) {
				t.Errorf("%s runs a scenario in %q, which is missing from e2eRegions in config/regions.go", filepath.Base(path), match[1])

e2e/test_helpers.go:405

• For Windows images, adding a region to e2eRegions does not fix this failure because replicationRegions ignores that list and returns only westus3. The diagnostic should direct users to the image OS's replication set instead; otherwise a Windows run in westus2 is told to add a region that is already present.

		t.Fatalf("scenario %q runs in region %q, which images are not replicated to; add the region to e2eRegions in e2e/config/regions.go",
			t.Name(), s.Location)

e2e/config/azure.go:667

• For a Windows image, this remediation is incorrect: replicationRegions ignores e2eRegions and returns only westus3. Point the error at the image OS's replication configuration so users are not told to modify a list that cannot make their run succeed.

		return fmt.Errorf("image %s is not replicated to %s: add the region to e2eRegions in e2e/config/regions.go, or run E2E in one of %s",
			image.Name, location, strings.Join(desired, ", "))