Skip to content

Add MAUI iOS Inner Loop measurements for CI#5187

Draft
davidnguyen-tech wants to merge 128 commits into
dotnet:mainfrom
davidnguyen-tech:nguyendav/maui-ios-inner-loop
Draft

Add MAUI iOS Inner Loop measurements for CI#5187
davidnguyen-tech wants to merge 128 commits into
dotnet:mainfrom
davidnguyen-tech:nguyendav/maui-ios-inner-loop

Conversation

@davidnguyen-tech

Copy link
Copy Markdown
Member

Summary

Adds MAUI iOS Inner Loop performance measurements to CI, supporting both iOS simulators and physical devices.

What's included:

  • iOSInnerLoopParser.cs — Binlog parser extracting iOS build task/target timings
  • Startup.cs/Reporter.cs — Wiring and null-safety fix
  • ioshelper.py — iOS simulator and physical device management
  • runner.py — IOSINNERLOOP execution branch (build → deploy → measure → incremental)
  • Scenario scripts — pre.py, test.py, post.py, setup_helix.py
  • maui_scenarios_ios_innerloop.proj — Helix workitem definition (simulator + physical device)
  • Pipeline YAML — Job entries in sdk-perf-jobs.yml and routing in run_performance_job.py

Measurements:

  • First deploy: full build + install + launch timing via binlog parsing
  • Incremental deploy: source edit → rebuild + reinstall + relaunch timing

Targets:

  • iOS Simulator (iossimulator-arm64) on macOS Helix machines
  • Physical iPhone (ios-arm64) on Mac.iPhone.17.Perf queue

Based on:

  • Existing Android Inner Loop CI scenario (working reference)
  • Local iOS implementation from feature/measure-maui-ios-deploy branch

davidnguyen-tech and others added 30 commits April 3, 2026 15:54
- Create iOSInnerLoopParser.cs: binlog parser for iOS inner loop build
  timings, extracting iOS-specific tasks (AOTCompile, Codesign, MTouch,
  etc.) and targets (_AOTCompile, _CodesignAppBundle, _CreateAppBundle,
  etc.) plus shared tasks (Csc, XamlC, LinkAssembliesNoShrink)

- Wire into Startup.cs: add iOSInnerLoop to MetricType enum and map it
  to iOSInnerLoopParser in the parser switch expression

- Fix Reporter.cs: guard against null/empty PERFLAB_BUILDTIMESTAMP to
  prevent ArgumentNullException on DateTime.Parse(null) when the env
  var is unset (falls back to DateTime.Now)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- const.py: Add IOSINNERLOOP constant and SCENARIO_NAMES mapping
- ioshelper.py: New module with iOSHelper class for simulator and physical
  device management (boot, install, launch, terminate, uninstall, find bundle)
- runner.py: Add iosinnerloop subparser, attribute assignment, and full
  execution branch (first build+deploy+launch, incremental loop with source
  toggling, binlog parsing, report aggregation, and Helix upload)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
pre.py: Install maui-ios workload, create MAUI template (no-restore for
Helix), strip non-iOS TFMs with flexible regex, inject MSBuild properties
(AllowMissingPrunePackageData, UseSharedCompilation), copy merged
NuGet.config for Helix-side restore, create modified source files for
incremental edit loop, check Xcode compatibility.

test.py: Thin entrypoint that builds TestTraits and invokes Runner.

post.py: Uninstall app from simulator, shut down dotnet build server,
clean directories.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Create the Helix machine setup script for MAUI iOS inner loop measurements.
This script runs on the macOS Helix machine before test.py and handles:

1. DOTNET_ROOT/PATH configuration from the correlation payload SDK
2. Xcode selection — auto-detects highest versioned Xcode_*.app, matching
   the pattern used by maui_scenarios_ios.proj PreparePayloadWorkItem
3. iOS simulator runtime validation via xcrun simctl
4. Simulator device boot with graceful already-booted handling
5. maui-ios workload install using rollback file from pre.py, with
   --ignore-failed-sources for dead NuGet feeds
6. NuGet package restore with --ignore-failed-sources /p:NuGetAudit=false
7. Spotlight indexing disabled via mdutil to prevent file-lock errors

Follows the same structure as the Android inner loop setup_helix.py:
context dict pattern, step-by-step functions, structured logging to
HELIX_WORKITEM_UPLOAD_ROOT for post-mortem debugging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Define the Helix .proj file for iOS inner loop measurements, modeled after
the Android inner loop .proj and existing maui_scenarios_ios.proj patterns.

Key design decisions:
- Build on Helix machine (not build agent) because deploy requires a
  connected device/simulator. PreparePayloadWorkItem only creates the
  template and modified source files via pre.py.
- Workload packs stripped from correlation payload (RemoveDotnetFromCorrelation
  Staging) and reinstalled on Helix machine by setup_helix.py.
- Environment variables set via shell 'export' in PreCommands (not in Python)
  because os.environ changes don't persist across process boundaries.
- No XHarness — iOS inner loop uses xcrun simctl directly.
- Simulator-only for now; physical device support (ios-arm64, code signing)
  is structured as a future TODO pending runner.py device support.
- 01:30 timeout to accommodate iOS build + workload install + NuGet restore.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- sdk-perf-jobs.yml: Add Mono Debug job entry for maui_scenarios_ios_innerloop
  on osx-x64-ios-arm64 (Mac.iPhone.17.Perf queue)
- run-performance-job.yml: Add maui_scenarios_ios_innerloop to the in() check
  so --runtime-flavor is forwarded to run_performance_job.py
- run_performance_job.py: Add maui_scenarios_ios_innerloop to
  get_run_configurations() (CodegenType, RuntimeType, BuildConfig) and to the
  binlog copy block for PreparePayloadWorkItems artifacts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ioshelper.py: Add detect_connected_device() with auto-detection via
  xcrun devicectl (JSON + fallback text parsing), uninstall_app_physical,
  terminate_app_physical, close_physical_device, and cleanup() dispatch
- runner.py: Add --device-type arg (simulator/device) to iosinnerloop
  subparser, auto-infer from RuntimeIdentifier, auto-detect device UDID,
  branch setup/install/startup/cleanup for physical vs simulator
- setup_helix.py: Detect device type from IOS_RID env var, skip simulator
  boot for physical device, add detect_physical_device() for Helix
- post.py: Handle physical device uninstall via devicectl with UDID
  auto-detection fallback
- maui_scenarios_ios_innerloop.proj: Add physical device HelixWorkItem
  (conditioned on iOSRid=ios-arm64), pass IOS_RID to Pre/PostCommands,
  add --device-type arg to both simulator and device workitems

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e sort

Fix 1 (Major): Replace non-existent 'devicectl terminate --bundle-id' with
'--terminate-existing' flag on launch command. Make terminate_app_physical()
a no-op with documentation explaining why.

Fix 2 (Medium): Write devicectl JSON output to temp file instead of
/dev/stdout, which mixes human-readable table and JSON. Applied in both
ioshelper.py and setup_helix.py with proper temp file cleanup.

Fix 3 (Medium): Add standard UUID pattern (8-4-4-4-12) to UDID regex in
_detect_device_fallback() for CoreDevice UUID format compatibility.

Fix 4 (Medium): Normalize MAUI template to always use Pages/ subdirectory
in pre.py. If template puts MainPage files at root, move them to Pages/.
Add explanatory comment in .proj documenting the coupling.

Fix 5 (Minor): Use tuple-of-ints version sort for Xcode selection instead
of string comparison (fixes 16.10 < 16.2 ordering bug).

Fix 6 (Minor): Make simulator boot failure fatal with sys.exit(1). Add
dynamic fallback to latest available iPhone simulator before failing.

Fix 7 (Nit): Add missing trailing newline to runner.py.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace deprecated tempfile.mktemp() with tempfile.mkstemp() in both
  ioshelper.py and setup_helix.py to avoid TOCTOU race condition.
- Fix unreachable fallback in detect_connected_device(): when devicectl
  exits non-zero (e.g., older Xcode without --json-output), call
  _detect_device_fallback() instead of returning None immediately.
- Guard against missing JSON report in runner.py IOSINNERLOOP branch:
  Startup.cs only writes reports when PERFLAB_INLAB=1, so local runs
  would crash with FileNotFoundError. Now degrades gracefully with
  empty counters and a warning.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Temporarily disable all other scenario jobs to speed up CI
iteration while validating the new MAUI iOS Inner Loop scenario.
This change should be reverted before merging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Capture dotnet build output instead of crashing on CalledProcessError
- Create traces/ directory before first build
- Fix setup_helix.py to write output.log (matches .proj expectation)
- Improve error handling for build failures

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet build stdout/stderr wasn't appearing in Helix console logs,
making it impossible to diagnose build failures. Explicitly capture and
print build output through Python logging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build 2943141 hit the 90-minute timeout. iOS first build with AOT
compilation can take 30+ minutes, plus 3 incremental iterations.
Increasing to 2.5 hours to allow full completion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Helix machines have Xcode 26.2 but the iOS SDK requires 26.3.
The minor version difference shouldn't affect build correctness,
so bypass the check with ValidateXcodeVersion=false.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mac.iPhone.17.Perf queue uses Intel x64 machines which need
iossimulator-x64, not iossimulator-arm64. Add architecture
detection in setup_helix.py and update default RID in .proj.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The traces upload directory already exists from the first build,
causing copytree() to fail on subsequent iterations. Clear it
before each parsetraces() call.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The -v:n flag was added to debug build errors but produces
excessive file copy logs. Default verbosity shows errors/warnings.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add /p:MtouchLink=None to disable managed linker for Debug inner
  loop builds, avoiding MT0180 errors on machines without Xcode 26.3
- Add minimum Xcode version check in setup_helix.py for fast failure
  with clear diagnostics when machine has Xcode < 26.0

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove temporary ${{ if false }}: wrappers that disabled all jobs
except iOS inner loop during iterative CI debugging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…loop

- Separate install from simulator/device setup in ioshelper.py
- Capture install time for first and incremental deploys in runner.py
- Add "Install Time" counter to both perf reports
- Add CoreCLR Debug job entry in pipeline YAML
- Add device (ios-arm64) job entries for both Mono and CoreCLR
- Wire iOSRid env var through to MSBuild for device builds
- [TEMP] Disable non-iOS-inner-loop jobs for CI validation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the dynamically-resolved manifest references SDK packs not yet
propagated to NuGet feeds, fall back to installing without the
rollback file. This avoids CI being blocked by transient feed
propagation delays.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the rollback file references SDK packs not yet propagated to
NuGet feeds, retry without the rollback file. Matches the fallback
pattern already added to pre.py.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The simulator HelixWorkItem was unconditionally included, even when
iOSRid=ios-arm64. This caused the simulator to receive device RID
in _MSBuildArgs, producing ARM64 binaries that can't install on a
simulator. Add Condition to exclude it from device jobs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t.py

Consolidate 12 duplicated simulator/device methods in ioshelper.py into
a unified API (setup_device, install_app, measure_cold_startup, cleanup)
that dispatches internally based on is_physical_device. Removes all
if-is_physical dispatch branches from runner.py.

Extract merge_build_deploy_and_startup and _make_counter to module-level
helpers. Inline the incremental iteration loop (was a nested function
with 10 parameters). Simplify post.py to reuse ioshelper instead of
duplicating device detection. Extract inject_csproj_properties in pre.py.

Net reduction: -232 lines across 4 files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ling

Re-apply run_env_vars (including iOSRid) right before perf_send_to_helix()
so the MSBuild .proj ItemGroup conditions can correctly exclude the
simulator work item from device jobs.

Add '|| exit $?' to setup_helix.py PreCommands so that when setup_helix
exits non-zero (e.g., Xcode too old), the Helix shell stops instead of
continuing to run test.py which would fail with a less clear error.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Env var inheritance through msbuild.sh/tools.sh is unreliable for
iOSRid. Add ios_rid field to PerfSendToHelixArgs and pass it as
/p:iOSRid=<value> on the MSBuild command line so it reaches .proj
evaluation deterministically. Also set it via set_environment_variables
as belt-and-suspenders.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mono_InnerLoop → Mono_InnerLoop_Simulator
CoreCLR_InnerLoop → CoreCLR_InnerLoop_Simulator

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace simctl (simulator) and devicectl (device) install/launch commands
with mlaunch to match the real Visual Studio F5 developer experience:

- Simulator: --launchsim combines install + launch (install_app returns 0)
- Device: --installdev for install, --launchdev for launch
- Device cleanup: --uninstalldevbundleid replaces devicectl uninstall
- Simulator cleanup: unchanged (simctl terminate + uninstall)
- Added _resolve_mlaunch() to find mlaunch from iOS SDK packs

Device detection (devicectl) and simulator management (simctl boot/
terminate/uninstall) remain unchanged. The install_app/measure_cold_startup
API is preserved so runner.py requires no changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of making install_app() a no-op for simulator, use
mlaunch --installsim to get a separate install measurement.
measure_cold_startup() still uses --launchsim for launch timing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…stall

After installing the maui-ios workload, read _RecommendedXcodeVersion
from the SDK's Versions.props and switch to the matching Xcode_*.app
if the currently active Xcode doesn't match. This handles the case
where Helix agents have a newer Xcode than the SDK requires.

Falls back gracefully to the already-selected Xcode if no matching
installation is found.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (7)

src/scenarios/shared/runner.py:1541

  • In the iOS inner-loop path you're already doing a single final TRACEDIR → Helix copy/upload at the end, but this call uses the default copy_traces=True. That causes an extra (and potentially large) directory copy per parse pass even though uploads are disabled here, increasing work-item time and I/O.
                startup.parsetraces(self.traits)

src/scenarios/shared/runner.py:1616

  • Per-iteration rmtree + startup.parsetraces() triggers a full TRACEDIR → Helix upload-dir copy on every incremental iteration. This is O(N) large directory copies and can dominate the inner-loop measurement runtime. The Android inner-loop path avoids this by using copy_traces=False and doing a single final copy at the end.
                    # Clear stale traces upload dir so copytree in parsetraces doesn't collide
                    helix_upload_dir = helixuploaddir()
                    if helix_upload_dir is not None:
                        traces_upload = os.path.join(helix_upload_dir, 'traces')
                        if os.path.exists(traces_upload):

src/scenarios/shared/runner.py:1652

  • The final incremental E2E report is written with only a tests array and drops any top-level metadata (build info, run info, inLab flags, etc.) that exists in the per-iteration Startup-generated reports. This can make the incremental report inconsistent with the first-deploy report and may affect PerfLab ingestion/classification.
                final_test = dict(report_template or {})
                final_test["counters"] = final_counters
                with open(incremental_e2e_report, 'w') as f:
                    json.dump({"tests": [final_test]}, f, indent=2)

src/scenarios/mauiiosinnerloop/post.py:18

  • This cleanup uses a bundle id derived from the MAUI template default (com.companyname...), but the scenario explicitly sets/passes net.dot.mauiiosinnerloop (see pre.py and the HelixWorkItem --bundle-id). As written, post.py will uninstall the wrong app id and leave the measured app installed on the simulator/device.
    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/sdk-perf-jobs.yml:20

  • This hard-coded if false disables the entire public correctness job section unconditionally, regardless of the runPublicJobs parameter. That changes CI behavior well beyond iOS inner-loop enablement and will mask regressions in tooling/tests for PRs.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

  • Same issue as above: this unconditional if false disables the public scenario benchmark jobs even when runPublicJobs is true. If the intent is to gate only the new iOS inner-loop work, this should not turn off unrelated public CI coverage.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:368

  • This if false disables the main private scenario benchmark matrix under runPrivateJobs. That effectively turns the pipeline into an iOS-only run and prevents detecting regressions in other scenario suites. If you need a temporary WIP gate, consider a dedicated parameter (or separate pipeline) rather than hard-coding false here.
  - ${{ if false }}: # disabled: iOS-only WIP test run
    - template: /eng/pipelines/templates/build-machine-matrix.yml

Build 3028071: the WWDR G3 intermediate was already present, yet codesign still
failed "unable to build chain to self-signed root". In the non-GUI Helix session
codesign isn't reaching the Apple Root anchor, so import the full chain (WWDR G3
+ Apple Root CA) into signing-certs.keychain-db, and log keychain/cert
diagnostics before signing to pinpoint any remaining trust gap.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 21, 2026 16:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (9)

src/scenarios/shared/runner.py:1487

  • Splitting --msbuild-args with a regex on whitespace/semicolons breaks quoted MSBuild arguments (e.g. /p:Foo="a b") and can change semantics or make dotnet build fail. Use a shell-like split that respects quoting, and only treat ';' as an additional separator.
            if self.msbuildargs:
                # _MSBuildArgs in the proj uses both ';' and ' ' as separators.
                for arg in re.split(r'[;\s]+', self.msbuildargs):
                    if arg.strip():
                        base_cmd.append(arg.strip())

src/scenarios/shared/runner.py:233

  • --csproj-path is enforced later via a runtime exception, but the CLI parser does not mark it as required. Making it required improves UX (immediate argparse error/help) and matches the Android inner loop subcommand behavior.
        iosinnerloopparser.add_argument('--csproj-path', help='Path to .csproj file to build', dest='csprojpath')

src/scenarios/shared/runner.py:239

  • --bundle-id is required for iOS inner loop runs (runner raises if missing), but it isn't marked required in argparse, which delays failures and makes the help text misleading.
        iosinnerloopparser.add_argument('--bundle-id', help='iOS bundle identifier', dest='bundleid')

src/scenarios/mauiiosinnerloop/post.py:18

  • The scenario sets the app ApplicationId to net.dot.mauiiosinnerloop (see pre.py / maui_scenarios_ios.proj), but post.py tries to uninstall com.companyname.<exename>. This mismatch means cleanup will generally not uninstall the measured app from the simulator/device.
    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

  • This script invokes the inner-loop scenario with bundle id com.companyname.mauiiosinnerloop, but the scenario’s csproj is explicitly rewritten to net.dot.mauiiosinnerloop (pre.py + Helix work items). If runOnAgent is ever enabled, install/launch will target the wrong bundle id and fail or measure the wrong app.
  --bundle-id com.companyname.mauiiosinnerloop \

eng/pipelines/sdk-perf-jobs.yml:21

  • This hard-disables the public correctness jobs by changing the condition to if false, which would eliminate CI coverage unrelated to iOS inner loop. If this PR is intended to add a scenario, it should not globally disable other job groups; keep the original gating and, if needed, add a separate feature flag for the new jobs.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP
  - job: Tooling_Tests_Windows

eng/pipelines/sdk-perf-jobs.yml:57

  • This if false disables the public scenario benchmark matrix entirely. That’s a broad CI behavior change that’s likely unintended for an iOS inner loop addition; re-enable the original parameter gating so PRs still get scenario coverage.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

  • This disables scheduled private jobs (runScheduledPrivateJobs) by forcing the condition to if false. That will stop the normal scheduled perf runs for all scenarios, which is a significant operational change unrelated to adding an iOS inner loop scenario.
- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:369

  • Large parts of the private job suite are wrapped in if false (e.g., scenario benchmarks, MAUI Android benchmarks, NativeAOT). This effectively turns the pipeline into an iOS-only run and removes broad perf coverage. Consider keeping existing private jobs enabled and only adding the iOS inner loop jobs behind their own condition/parameter.
  - ${{ if false }}: # disabled: iOS-only WIP test run
    - template: /eng/pipelines/templates/build-machine-matrix.yml
      parameters:

…or is used

Diagnostics (build 3028084) proved the full chain is present + correctly linked
(openssl: leaf -> WWDR G3 -> Apple Root, AKID==SKID), yet codesign still failed
"unable to build chain to self-signed root". Cause: `--keychain
signing-certs.keychain-db` scopes trust evaluation to that keychain, where Apple
Root is present but not a trusted anchor. Sign without --keychain first so trust
uses the System keychain's built-in Apple Root anchor (identity still resolved
from the search list); keep the keychain-scoped form as a fallback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 21, 2026 17:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (5)

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

  • Bundle identifier here doesn’t match the ApplicationId forced in pre.py (net.dot.mauiiosinnerloop) nor the Helix .proj invocation. Using a different bundle id will make install/launch/uninstall operations target the wrong app and fail or skew measurements.
  --bundle-id com.companyname.mauiiosinnerloop \

src/scenarios/mauiiosinnerloop/post.py:18

  • post.py uninstalls com.companyname., but the scenario sets ApplicationId to net.dot.mauiiosinnerloop in pre.py and uses that bundle id during the run. This cleanup currently targets the wrong bundle id, so the app can be left installed and interfere with later runs.
    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/sdk-perf-jobs.yml:20

  • This hard-coded if false permanently disables all public correctness/scenario jobs regardless of the pipeline parameters. If this is meant to be temporary WIP gating, it should be driven by parameters (or removed) so normal CI behavior isn’t silently turned off.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

  • This second hard-coded if false permanently disables the public scenario benchmark job group, regardless of the runPublicJobs parameter.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

  • This hard-coded if false disables the scheduled private job set entirely, regardless of the runScheduledPrivateJobs parameter. That changes pipeline behavior beyond adding the new iOS inner loop scenario.
- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

Comment on lines +17 to +18
public void EnableKernelProvider(ITraceSession kernel) { throw new NotImplementedException(); }
public void EnableUserProviders(ITraceSession user) { throw new NotImplementedException(); }
…chain

Device codesign still failed "unable to build chain to self-signed root" with
the full chain present in signing-certs.keychain-db and with/without --keychain,
i.e. the self-signed Apple Root isn't treated as a trusted anchor in the
headless Helix session. Work around without sudo: also import WWDR G3 + Apple
Root into the login keychain (which SecTrust consults), and add the Apple Root
as a trusted code-signing anchor in the user trust domain (add-trusted-cert -r
trustRoot -p codeSign). All best-effort + timeout-guarded so the work item can't
hang.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 21, 2026 18:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (7)

src/scenarios/shared/runner.py:237

  • The iOS inner loop subcommand defers required-argument validation to runtime exceptions. This is inconsistent with other subcommands (e.g. androidinnerloop) and makes CLI usage/error messages worse. Mark the required arguments as required=True and enforce a valid iteration range via argparse (1+).
        iosinnerloopparser.add_argument('--csproj-path', help='Path to .csproj file to build', dest='csprojpath')
        iosinnerloopparser.add_argument('--edit-src', help='Modified source file paths, semicolon-separated', dest='editsrc')
        iosinnerloopparser.add_argument('--edit-dest', help='Destination paths for modified files, semicolon-separated', dest='editdest')
        iosinnerloopparser.add_argument('--framework', '-f', help='Target framework (e.g., net11.0-ios)', dest='framework')
        iosinnerloopparser.add_argument('--configuration', '-c', help='Build configuration', dest='configuration', default='Debug')

src/scenarios/mauiiosinnerloop/post.py:18

  • Bundle ID used for cleanup doesn't match the bundle ID configured during scenario setup (pre.py sets ApplicationId to net.dot.mauiiosinnerloop). As-is, post cleanup will skip uninstalling the actual app, and simulator/device state can leak across work items.
    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

  • This on-agent runner passes --bundle-id com.companyname.mauiiosinnerloop, but the scenario scaffolding sets the MAUI ApplicationId to net.dot.mauiiosinnerloop (pre.py). With the current value, simulator launch/measurement will fail because simctl/devicectl lookups use the wrong bundle id.
  --bundle-id com.companyname.mauiiosinnerloop \

eng/pipelines/sdk-perf-jobs.yml:20

  • parameters.runPublicJobs is replaced with if false, which disables all public correctness and scenario coverage regardless of pipeline parameters. This looks like a WIP workaround, but it would prevent the normal CI gates from running if merged.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

  • This block is also guarded by if false, so public scenario jobs never run even when runPublicJobs: true. If the intent is to keep existing behavior, restore the parameter-based condition.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:367

  • This if false wrapper disables the main private scenario benchmark matrix even when runPrivateJobs: true. If this is only for temporary experimentation, consider using a dedicated parameter; otherwise restore the original behavior so private CI coverage isn't silently removed.
  - ${{ if false }}: # disabled: iOS-only WIP test run

eng/pipelines/sdk-perf-jobs.yml:826

  • Scheduled private jobs are permanently disabled via if false, so setting runScheduledPrivateJobs: true will have no effect. Restore the parameter-based condition so scheduled runs can execute as configured.
- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

…text)

Build 3028131 gave the definitive diagnosis: add-trusted-cert failed with
"SecTrustSettingsSetTrustSettings: The authorization was denied since no user
interaction was possible" — the device work item runs in a non-interactive
session, so Security trust operations (and codesign's chain-to-anchor
evaluation) are denied. XHarness avoids this by running under `launchctl asuser`.

Add an attempt to run codesign via `sudo -n launchctl asuser <uid>` (a real user
session) before the plain forms. Uses `sudo -n` + a subprocess timeout so it
fails fast / never hangs if not permitted.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 21, 2026 18:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (7)

src/scenarios/shared/runner.py:1652

  • The final incremental E2E report is written as { "tests": [...] } only, which drops the top-level build/os/run metadata that the Startup tool’s Reporter normally includes. This will likely cause PerfLab uploads to be rejected or misclassified (see the Android inner loop path which preserves top-level metadata). Preserve the top-level fields from an existing report (e.g., the first-build report) when writing the aggregated incremental report.
                final_test = dict(report_template or {})
                final_test["counters"] = final_counters
                with open(incremental_e2e_report, 'w') as f:
                    json.dump({"tests": [final_test]}, f, indent=2)

src/scenarios/mauiiosinnerloop/post.py:18

  • post.py uninstalls using com.companyname.<exename> but the scenario’s ApplicationId is set to net.dot.mauiiosinnerloop in pre.py and the Helix work item runs with --bundle-id net.dot.mauiiosinnerloop. This mismatch means cleanup will usually fail to uninstall the measured app, leaving state behind for subsequent runs.
    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/sdk-perf-jobs.yml:20

  • This changes the public-jobs gating from parameters.runPublicJobs to if false, which permanently disables public correctness jobs and public scenario coverage. That’s a major CI behavior change and doesn’t align with the PR goal of adding iOS inner loop measurements to CI.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

  • This second if false block also permanently disables the rest of the public-jobs section. If the intent is to temporarily reduce CI surface while iterating, consider gating only the new iOS inner loop jobs rather than disabling all public jobs.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

  • Scheduled private jobs are currently hard-disabled via if false, which prevents scheduled perf coverage from running at all. If this PR is intended to land, this should remain controlled by parameters.runScheduledPrivateJobs rather than an unconditional false.
- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:367

  • Large portions of the private-jobs matrix are also hard-disabled via if false (starting here). That effectively turns the pipeline into an iOS-only run, which is a substantial behavioral change beyond this PR’s stated scope.
  - ${{ if false }}: # disabled: iOS-only WIP test run

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

  • The on-agent (dormant) inner loop driver uses --bundle-id com.companyname.mauiiosinnerloop, but the scenario’s csproj is rewritten to use ApplicationId=net.dot.mauiiosinnerloop and the Helix work items use that bundle id. Keeping this consistent avoids confusing uninstall/launch failures if this path is enabled later.
  --bundle-id com.companyname.mauiiosinnerloop \

Comment on lines +839 to +849
collect_cmd = ['sudo', 'log', 'collect', '--device',
'--start', start_ts, '--output', logarchive]
RunCommand(collect_cmd, verbose=True).run()

# Parse SpringBoard watchdog events for this bundle ID.
# sudo because the logarchive is root-owned (created by sudo log collect).
show_cmd = ['sudo', 'log', 'show',
'--predicate', '(process == "SpringBoard") && (category == "Watchdog")',
'--info', '--style', 'ndjson', logarchive]
show = RunCommand(show_cmd, verbose=True)
show.run()
…p clean failure

The `sudo -n launchctl asuser` codesign attempt (build 3028142) confirmed asuser
IS permitted, but codesign hung in that session because the keychain isn't
unlocked there, and the fallback then hit "Permission denied". Revert to the
plain codesign path so the device job fails fast with the clear diagnostic, and
drop the add-trusted-cert (proven denied: "no user interaction was possible").
The real fix is running the device work item under launchctl asuser like
XHarness — infra, not scenario code.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 21, 2026 18:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (8)

src/scenarios/mauiiosinnerloop/post.py:18

  • post.py computes bundle_id as com.companyname., but pre.py explicitly sets to net.dot.mauiiosinnerloop and runner/test invocations use that value. Using the wrong bundle id means uninstall/terminate targets the wrong app and leaves the measured app installed (affecting subsequent runs).
    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

  • run-ios-innerloop-on-agent.sh passes --bundle-id com.companyname.mauiiosinnerloop, but the scenario pre.py forces ApplicationId to net.dot.mauiiosinnerloop (required for device signing and used by Helix runs). This mismatch will cause install/launch/uninstall to target the wrong bundle id on agent runs.
  --bundle-id com.companyname.mauiiosinnerloop \

src/scenarios/shared/runner.py:1542

  • StartupWrapper.parsetraces() defaults to copying TRACEDIR into the Helix upload dir every call. In the inner-loop scenario this is invoked repeatedly while TRACEDIR grows (multiple binlogs), which creates O(N^2) I/O and can significantly slow the work item. Pass copy_traces=False here and do the single final copy/upload once at the end (which this code already does).
                                       scenarioname=scenarioprefix + " - First Build and Deploy",
                                       upload_to_perflab_container=False)
                startup.parsetraces(self.traits)

src/scenarios/shared/runner.py:1619

  • Same as the first-deploy parse: calling StartupWrapper.parsetraces() per iteration with the default copy_traces=True repeatedly copies an ever-growing TRACEDIR into the Helix upload dir, which is avoidable overhead in a tight loop. Use copy_traces=False and rely on the single final copy/upload after the loop.
                        if os.path.exists(traces_upload):
                            rmtree(traces_upload)

                    startup.parsetraces(self.traits)

eng/pipelines/sdk-perf-jobs.yml:21

  • This changes the public-jobs gate from parameters.runPublicJobs to a hardcoded false, which will permanently disable the public correctness job block. If this PR is meant to add iOS inner-loop measurements without turning off unrelated CI, revert this condition back to the parameter gate.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP
  - job: Tooling_Tests_Windows

eng/pipelines/sdk-perf-jobs.yml:60

  • This gate is also hardcoded to false, so the public scenario-benchmark jobs will never run even when parameters.runPublicJobs is true. If this is only intended as temporary experimentation, it should not be merged as-is; otherwise restore the original parameter-based condition.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

  # Scenario benchmarks
  - template: /eng/pipelines/templates/build-machine-matrix.yml

eng/pipelines/sdk-perf-jobs.yml:829

  • The scheduled-private-jobs gate is hardcoded to false, which disables all scheduled runs regardless of the pipeline parameters. This looks unrelated to adding iOS inner-loop measurements and will stop existing scheduled perf coverage; restore the original parameter-based condition.
- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

  # SDK scenario benchmarks
  - template: /eng/pipelines/templates/build-machine-matrix.yml

eng/pipelines/sdk-perf-jobs.yml:369

  • Large sections of the private job matrix are wrapped in if false blocks (e.g., Scenario benchmarks, affinitized runs, Maui Android, NativeAOT). That means enabling parameters.runPrivateJobs will no longer run the existing private perf coverage, which is a significant CI behavior change unrelated to adding a new iOS scenario. Consider reverting these if false gates (or scoping any temporary disablement to the new iOS inner-loop jobs only).
- ${{ if parameters.runPrivateJobs }}:

  # Scenario benchmarks
  - ${{ if false }}: # disabled: iOS-only WIP test run
    - template: /eng/pipelines/templates/build-machine-matrix.yml
      parameters:

…+ codesign together)

The device work item's non-interactive session denies codesign's trust
evaluation ("no user interaction was possible"); `launchctl asuser` is permitted
but a bare asuser codesign hung because the keychain wasn't unlocked in that
session. Generate a script that unlocks signing-certs.keychain-db AND deep-signs
every Mach-O/.app/.framework, and run the whole script in one `sudo -n launchctl
asuser <uid>` invocation so unlock + codesign share the user session (the
keychain password is read from ~/.config/keychain at runtime, never embedded).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 21, 2026 19:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:176

  • pre.py forces the MAUI template's ApplicationId to net.dot.mauiiosinnerloop, but this on-agent driver launches with --bundle-id com.companyname.mauiiosinnerloop. That bundle id mismatch will make simctl launch fail (and cleanup may target the wrong app). Use the same bundle id as the scenario payload and Helix work items.
python3 test.py iosinnerloop \
  --csproj-path app/MauiiOSInnerLoop.csproj \
  --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" \
  --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" \
  --bundle-id com.companyname.mauiiosinnerloop \
  -f "$framework-ios" -c Debug \

src/scenarios/mauiiosinnerloop/post.py:20

  • Cleanup is uninstalling com.companyname.<exename> but the scenario sets the app ApplicationId to net.dot.mauiiosinnerloop (see pre.py and the Helix work item --bundle-id). With the current value, uninstall/terminate will be a no-op and can leave the tested app installed/running across work items. Consider keeping post.py in sync with the scenario’s bundle id.
try:
    bundle_id = f'com.companyname.{EXENAME.lower()}'
    ios_rid = os.environ.get('IOS_RID', 'iossimulator-arm64')
    is_physical = (ios_rid == 'ios-arm64')

eng/pipelines/sdk-perf-jobs.yml:21

  • This hard-disables the entire public correctness job set (tooling tests + public scenario runs) by changing the template condition to if false. That will silently stop CI coverage unrelated to iOS inner loop. If the intent is temporary, gate it behind the existing runPublicJobs parameter (or a new explicit parameter), but don’t merge with a constant-false condition.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP
  - job: Tooling_Tests_Windows

eng/pipelines/sdk-perf-jobs.yml:827

  • This hard-disables scheduled private runs by replacing the runScheduledPrivateJobs gate with if false, which prevents any scheduled performance coverage from running. This should remain gated by the runScheduledPrivateJobs parameter (or another explicit parameter), not a constant-false.
# Scheduled runs will run all of the jobs on the PerfTigers, as well as the Arm64 job
- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:369

  • Within the private-jobs section, this if false wrapper disables the standard cross-platform scenario benchmark matrix, leaving only the iOS WIP jobs. If the goal is to temporarily narrow coverage, consider using a parameter/variable-based condition; otherwise this should be restored so private runs still execute the normal scenario suite.
- ${{ if parameters.runPrivateJobs }}:

  # Scenario benchmarks
  - ${{ if false }}: # disabled: iOS-only WIP test run
    - template: /eng/pipelines/templates/build-machine-matrix.yml
      parameters:

…o (machine gap)

Build 3028196 ran unlock+codesign inside one `launchctl asuser` session (no hang
this time), but codesign STILL failed "unable to build chain to self-signed
root". So the session context is not the differentiator — even in a user session
with the full chain present in the keychains, codesign can't reach a trusted
Apple Root anchor. That's a code-signing trust-store provisioning gap on
Mac.iPhone.13.Perf, not scenario code. Revert to the clean XHarness-matching
codesign (--keychain + entitlements, deep, deepest-first).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 21, 2026 19:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

src/scenarios/mauiiosinnerloop/post.py:18

  • Bundle ID used for uninstall/cleanup doesn't match the bundle ID configured in pre.py (net.dot.mauiiosinnerloop) and passed to the scenario runner. This will leave the app installed between runs and can skew first-deploy measurements.
    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:176

  • The on-agent driver passes bundle ID com.companyname.mauiiosinnerloop, but the scenario's generated csproj is configured with ApplicationId net.dot.mauiiosinnerloop (pre.py). Using the wrong bundle ID will fail install/launch and/or report timings for a different app.
python3 test.py iosinnerloop \
  --csproj-path app/MauiiOSInnerLoop.csproj \
  --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" \
  --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" \
  --bundle-id com.companyname.mauiiosinnerloop \
  -f "$framework-ios" -c Debug \

eng/pipelines/sdk-perf-jobs.yml:20

  • This hard-codes public jobs to never run, ignoring the runPublicJobs parameter. That changes pipeline behavior beyond this scenario and can hide regressions in unrelated tooling/tests.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

  • This hard-codes public scenario jobs to never run, ignoring the runPublicJobs parameter and effectively disabling CI coverage for public PRs.
- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

  • This hard-codes scheduled private jobs to never run, ignoring the runScheduledPrivateJobs parameter and disabling scheduled performance coverage.
- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants