Add benchmark tracking on every push (gate PRs on allocation regressions) - #22
Merged
Conversation
Runs BenchmarkDotNet on every push to master and on PRs, then feeds per-benchmark allocated-bytes into benchmark-action/github-action-benchmark: - push to master records the new baseline on the gh-pages data branch (dev/bench); - PRs compare against that baseline, comment on a >10% allocation jump, and fail the check. Gates on allocated bytes rather than time: allocation counts are deterministic and stable on shared runners, so they can safely fail a build; time stays informational (in the run logs). JSON export -> jq reshape -> customSmallerIsBetter was validated locally against ScanBenchmark. Mirrors ci.yml's SDK/cache setup. The micro-benchmark filters activate automatically once PR #21 lands them on master.
guy-lud
added a commit
that referenced
this pull request
Jul 12, 2026
…23) * Session wrap: refresh handoff + fix-plan; gitignore BenchmarkDotNet output - Handoff/fix-plan now reflect #21 merged (Q1–Q4 + M1 collision fix + micro-benchmarks, proven 2.7x–32x on repeated paths) and #22 open (per-push benchmark tracking, gate on allocation regressions). - Records the durable facts: gate on allocations not time, gh-pages holds the baseline, and the M1 rule (generated impl name is separate from the section name). - Next priority remains P3. - gitignore BenchmarkDotNet.Artifacts/ so local benchmark runs don't leave untracked output. * gitignore .claude/settings.local.json (personal SessionStart hook)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Saves BenchmarkDotNet results on every push and surfaces the diff on PRs, so a perf/allocation regression is caught before it lands.
How it works
master, PRs tomaster, and manualworkflow_dispatch.dotnet runthe benchmark project (ShortRun) with the JSON exporter, filtered toScanBenchmark+ the Q1/Q3/Q4 micro-benchmarks.jqreshapes BenchmarkDotNet's*-report-full-compressed.jsonintogithub-action-benchmark'scustomSmallerIsBetterformat —[{name, unit: "bytes", value}].benchmark-action/github-action-benchmark@v1:gh-pagesdata branch (dev/bench).Key design choice: gate on allocations, not time
Allocated bytes are deterministic, so they're stable on shared GitHub runners and safe to fail a build on. Wall-clock time is far too noisy to gate — it stays in the run logs for eyeballing only. (This is also why the macro
ScanBenchmarktime didn't move for the quick wins but the micro-benchmark allocations did.)Notes for review
gh-pagesbranch (data only,dev/bench/) on the first master run — that'sgithub-action-benchmark's standard store. If you'd rather not have agh-pagesbranch, I can switch toexternal-data-json-path+actions/cacheinstead. Your call.ScanBenchmarkis tracked. No failure in the meantime.ScanBenchmark(2000 types) + 3 micro-benchmarks ≈ a few minutes per run;concurrencycancels superseded runs.ScanBenchmarkbefore authoring.