Skip to content

[Disk CI benchmarks] Use 1ES.Pool=diskann-github#869

Merged
arrayka merged 16 commits into
mainfrom
u/arrayka/1es-pool
May 20, 2026
Merged

[Disk CI benchmarks] Use 1ES.Pool=diskann-github#869
arrayka merged 16 commits into
mainfrom
u/arrayka/1es-pool

Conversation

@arrayka
Copy link
Copy Markdown
Contributor

@arrayka arrayka commented Mar 30, 2026

Switch .github/workflows/disk-benchmarks.yml and .github/workflows/disk-benchmarks-aa.yml to premium self-hosted VMs (L8s v3 Azure SKU with premium SSDs) to reduce noise and make performance gates more stable.

After this change, disk-based graph search throughput improved significantly - from 92 QPS to 596 QPS:

Before:
=======================================================================================================================================================
 L KNN      QPS  Mean Latency   95% Latency  99.9 Latency    IOs    IO (us)   CPU (us)   PQ Preprocess (us)  Mean Comps  Mean Hops  Cache Hit %  Recall
=======================================================================================================================================================
200 100     92.1     43391.1us       61602us      198334us  209.9  41693.3us   1657.8us               40.0us      5811.1      209.9         0.0%  91.506

After:
=======================================================================================================================================================
 L KNN      QPS  Mean Latency   95% Latency  99.9 Latency    IOs    IO (us)   CPU (us)   PQ Preprocess (us)  Mean Comps  Mean Hops  Cache Hit %  Recall
=======================================================================================================================================================
200 100    596.9      6687.5us        9923us       16727us  209.8   5406.7us   1248.3us               32.5us      5832.5      209.8         0.0%  91.428

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Mar 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.60%. Comparing base (d516da1) to head (abeab51).
⚠️ Report is 13 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #869      +/-   ##
==========================================
- Coverage   90.60%   90.60%   -0.01%     
==========================================
  Files         461      459       -2     
  Lines       85559    85482      -77     
==========================================
- Hits        77525    77447      -78     
- Misses       8034     8035       +1     
Flag Coverage Δ
miri 90.60% <ø> (-0.01%) ⬇️
unittests 90.56% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 65 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@arrayka arrayka changed the title Use 1ES.Pool=diskann-github [Disk CI benchmarks] Use 1ES.Pool=diskann-github May 18, 2026
@arrayka arrayka marked this pull request as ready for review May 18, 2026 20:44
@arrayka arrayka requested review from a team and Copilot May 18, 2026 20:44
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR moves disk benchmark workflows to the diskann-github 1ES self-hosted runner pool and adds local NVMe setup for benchmark workspace/data placement.

Changes:

  • Switches disk benchmark and A/A benchmark jobs from ubuntu-latest to self-hosted 1ES runners.
  • Adds NVMe formatting/mounting plus workspace symlinks in both benchmark workflows.
  • Updates the shared disk benchmark setup action to install Rust if rustup is missing.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
.github/workflows/disk-benchmarks.yml Runs macro benchmarks on the 1ES self-hosted pool and stages workspace paths on NVMe.
.github/workflows/disk-benchmarks-aa.yml Runs scheduled A/A benchmarks on the 1ES self-hosted pool and stages workspace paths on NVMe.
.github/actions/setup-disk-benchmark/action.yml Adds conditional Rust/rustup installation before showing the toolchain and downloading datasets.
Comments suppressed due to low confidence (2)

.github/workflows/disk-benchmarks.yml:87

  • These symlink creations are not idempotent. Self-hosted runner workspaces can persist between jobs, so a rerun or second matrix job on the same runner will fail here if diskann_rust or baseline already exists; remove/replace the links or use an idempotent workspace setup.
          ln -s /mnt/nvme/diskann_rust diskann_rust
          ln -s /mnt/nvme/baseline baseline

.github/workflows/disk-benchmarks-aa.yml:64

  • These symlink creations are not idempotent. Self-hosted runner workspaces can persist between jobs, so a rerun or second matrix job on the same runner will fail here if diskann_rust or baseline already exists; remove/replace the links or use an idempotent workspace setup.
          ln -s /mnt/nvme/diskann_rust diskann_rust
          ln -s /mnt/nvme/baseline baseline

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/disk-benchmarks.yml
Comment thread .github/workflows/disk-benchmarks.yml
Comment thread .github/workflows/disk-benchmarks-aa.yml
Comment thread .github/actions/setup-disk-benchmark/action.yml
@arrayka arrayka enabled auto-merge (squash) May 19, 2026 21:46
@arrayka arrayka linked an issue May 19, 2026 that may be closed by this pull request
Comment thread .github/actions/setup-disk-benchmark/action.yml
Copy link
Copy Markdown
Contributor

@arkrishn94 arkrishn94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just flagged one concern raised by Copilot. Want to make sure it is a non-issue?

Also, just out of curiosity - does the CI for this PR exercise these workflows?

Comment thread .github/workflows/disk-benchmarks.yml
@arrayka arrayka merged commit fd37402 into main May 20, 2026
23 checks passed
@arrayka arrayka deleted the u/arrayka/1es-pool branch May 20, 2026 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Switch performance gates to a more powerful agent pool

6 participants