perf(ci): parallelize scheduled Miri and reduce resource outliers - #706
Open
st-dev-gh wants to merge 7 commits into
Open
perf(ci): parallelize scheduled Miri and reduce resource outliers#706st-dev-gh wants to merge 7 commits into
st-dev-gh wants to merge 7 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes the scheduled full-workspace Miri runtime-analysis runs by compiling once and then executing test artifacts concurrently, while also reducing a few known Miri resource outliers (CPU and memory) to improve job stability and throughput.
Changes:
- Adds a shared PowerShell-based Miri runner that builds once (
--no-run), discovers test artifacts from Cargo JSON output, and runs them concurrently with per-artifact log grouping and telemetry. - Reduces Miri-only workload in a few high-cost test suites (Routerama deep trie, Multitude arena drop-slice breadth, and Multitude macros pretty-printing).
- Adds a strict-provenance-only ignore for an Internity test that triggers an unsupported dependency behavior on Unix.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| justfiles/anvil/checks/miri.just | Introduces _anvil-miri-test to compile once, run artifacts in parallel, and emit resource telemetry. |
| justfiles/anvil/checks/miri-tree-borrows.just | Switches tree-borrows profile to use the shared parallel runner. |
| justfiles/anvil/checks/miri-strict-provenance.just | Switches strict-provenance profile to use the shared parallel runner. |
| justfiles/anvil/checks/miri-race-coverage.just | Switches race-coverage profile to use the shared parallel runner. |
| crates/routerama/src/dyn_builder.rs | Reduces deep-trie segment count under Miri to bound worst-case CPU time. |
| crates/multitude/tests/arena.rs | Ignores certain runtime-boundary tests under Miri and reduces oversized drop-slice sizes under Miri to lower peak memory. |
| crates/multitude_macros_impl/src/lib.rs | Avoids formatting-only pretty-print work under Miri by comparing normalized token streams instead. |
| crates/multitude_macros_impl/Cargo.toml | Removes prettyplease dev-dependency no longer needed by tests. |
| crates/internity/tests/basic.rs | Ignores one test only under strict-provenance Miri due to an unsupported dependency cast path on Unix. |
| Cargo.lock | Updates lockfile to reflect removal of the prettyplease dependency. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #706 +/- ##
=======================================
Coverage 100.0% 100.0%
=======================================
Files 563 563
Lines 61068 61068
=======================================
Hits 61068 61068
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Evgenii (Vaiz)
requested changes
Aug 28, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
crates/internity/tests/basic.rs:596
- This
cfg_attrignores the test whenevermiri_strict_provenanceis set, even in non-Miri native builds that might also define that cfg. If the intent is to ignore only for strict-provenance Miri, gate it withall(miri, miri_strict_provenance)so native coverage remains unaffected by an incidental cfg flag.
#[cfg_attr(
miri_strict_provenance,
ignore = "parking_lot_core uses integer-to-pointer casts on Unix, which strict-provenance Miri rejects"
)]
Evgenii (Vaiz)
approved these changes
Aug 28, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
justfiles/anvil/checks/miri.just:166
Get-MemoryTelemetryforces a full GC + finalizer drain every time an artifact starts/completes. With dozens/hundreds of artifacts this can add substantial overhead and also perturbs the very resource profile you're trying to observe (collections can suspend all managed threads), which undercuts the goal of measuring Miri concurrency/resource outliers.
Consider treating the GC stats as passive telemetry here: read GetGCMemoryInfo() without forcing a collection. You already run one collection before starting the parallel work.
function Get-MemoryTelemetry {
[GC]::Collect()
[GC]::WaitForPendingFinalizers()
$info = [GC]::GetGCMemoryInfo()
Compile each profile once to preserve workspace feature unification, then run its test artifacts with bounded parallelism and longest-first scheduling. Avoid formatting-only macro test work under Miri while retaining normal syntax validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d9192741-03aa-4e35-aaae-5c58ddc3975b
Select one concurrent test artifact per logical processor and log GC memory telemetry so scheduled runs can reveal whether memory pressure limits the CPU-based configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d9192741-03aa-4e35-aaae-5c58ddc3975b
Preserve native stress coverage while bounding Routerama depth and high-cardinality Arena tests under Miri. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d9192741-03aa-4e35-aaae-5c58ddc3975b
Keep the concurrency test in native and other Miri profiles while excluding the Unix parking_lot path that strict provenance cannot interpret. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d9192741-03aa-4e35-aaae-5c58ddc3975b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d9192741-03aa-4e35-aaae-5c58ddc3975b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d9192741-03aa-4e35-aaae-5c58ddc3975b
st-dev-gh
force-pushed
the
u/st-dev-gh/anvil-scheduled
branch
2 times, most recently
from
August 28, 2026 12:40
f1904f4 to
61ea2e8
Compare
|
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
Suppressed comments (1)
crates/routerama/src/dyn_builder.rs:154
- The
#[cfg(test)] mod testsblock in this file is missing the repo-standard#[cfg_attr(coverage_nightly, coverage(off))]annotation used to keep unit-test-only lines out of the coverage gate (e.g. crates/tick/src/error.rs:103-105). Since this PR modifies the test module, please add the attribute above the module so coverage instrumentation focuses on non-test code.
#[cfg(miri)]
const DEEP_TRIE_SEGMENTS: usize = 512;
#[cfg(not(miri))]
const DEEP_TRIE_SEGMENTS: usize = 4_096;
martin-kolinek
approved these changes
Aug 28, 2026
|
|
||
| # Run selected Miri test artifacts with CPU-based parallelism. | ||
| [script("pwsh")] | ||
| _anvil-miri-test: |
Collaborator
There was a problem hiding this comment.
I guess this is intentional, but I really would like to already see an accompanying PR into cargo-anvil (or at least a cargo tool that does this).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Scheduled runtime analysis runs four full-workspace Miri profiles:
The profiles intentionally remain sequential, but
cargo miri testalso ran every test artifact sequentially within each profile. This left most runner CPUs idle and made a few extreme artifacts dominate runtime and memory consumption.This builds on #674, which optimizes the PR stacked-borrows job. That PR deliberately leaves the scheduled full-workspace profiles unchanged; this PR targets those scheduled runs.
Findings
A local Tree Borrows profile compiled the workspace once and measured all 226 test artifacts with four concurrent workers.
Most artifacts were small:
A few artifacts dominated resource use:
routeramamultitude::arenabytesbufarena_vecArena profiling isolated the largest consumers:
u16::MAX + 1Drop-slice tests reached over 20 GiB when run separately.Runtime and memory were correlated across ordinary artifacts, but the extreme outliers were different problems: Routerama was CPU-bound while Arena was memory-bound.
Changes
Compile once and execute artifacts concurrently
The shared Miri recipe now:
--no-run.*_macros_implcode-generation crates; emitted code remains covered through consuming crates.compiler-artifactoutput.cargo-miri runner.ANVIL_MIRI_JOBSoverride.Compiling the selected workspace together is important: invoking Miri separately per package changes Cargo feature unification and previously broke packages relying on workspace-unified features.
The four Miri profiles themselves remain sequential.
Add resource telemetry
Scheduled logs now report:
The telemetry is informational and does not silently reduce concurrency.
Bound Routerama's deep-trie workload
failed_build_discards_deep_source_trie_iterativelyretains 4,096 segments natively but uses 512 under Miri.This preserves the full native stack-safety stress test while still exercising deep trie construction, conflict traversal, and iterative destruction under Miri.
Measured Tree Borrows runtime fell from the multi-hour outlier to approximately 5.7 minutes.
Reduce Arena's Miri memory pressure
Tests whose only purpose is validating runtime length boundaries above
u16::MAXare ignored under Miri while retaining their full native coverage.Oversized Drop-slice tests still exercise the same allocation and destructor paths, but under Miri they use:
The complete Arena artifact now measures:
The excluded cases remain covered at their original sizes by native tests.
Exclude an unsupported strict-provenance dependency path
On Linux and Linux ARM, strict-provenance Miri rejected an integer-to-pointer conversion inside
parking_lot_corewhile running:internity::freeze_races_writer_and_stays_prefix_consistentThis is a platform-specific dependency implementation rather than an Internity provenance violation. The test is now ignored only for strict-provenance Miri and remains covered by:
Results
The latest Windows scheduled runtime-analysis job completed successfully.
Linux and Linux ARM also completed the optimized Arena and Routerama artifacts without runner loss or memory termination. They subsequently exposed the separate
parking_lot_corestrict-provenance incompatibility addressed by this PR.Coverage trade-offs
*_macros_implcrates are excluded because their tests parse and compare generated tokens rather than execute runtime code; emitted code remains covered by consuming crates.