[Intel NPU/GPU] Add Windows & Linux Intel NPU & GPU support by Looong01 · Pull Request #1171 · lightvector/KataGo

Looong01 · 2026-03-16T18:49:27Z

Summary

This PR adds and hardens the Windows & Linux Intel NPU path for KataGo using the ONNX backend with ONNX Runtime + OpenVINO Execution Provider, and updates docs/config guidance for an end-to-end workflow.

It also improves failure behavior for non-ONNX builds and simplifies Windows & Linux dependency handling.

What Changed

1) ONNX backend and OpenVINO provider support

Added/updated ONNX Runtime provider selection via onnxProvider (cpu, openvino, cuda, tensorrt, migraphx, coreml).
Added/updated OpenVINO-specific runtime options:
- onnxOpenVINODeviceType
- onnxOpenVINODeviceId
- onnxOpenVINOCacheDir
- onnxOpenVINOEnableNPUFastCompile (best-effort; depends on ORT build support)
Supports both:
- loading raw .onnx models directly
- loading .bin/.bin.gz models via internal conversion to ONNX graph

2) `exportonnx` command behavior

exportonnx is available in ONNX builds and exports fixed-size ONNX models.
Default export board size is 19x19 (-x/-y can override).
In non-ONNX builds, exportonnx now returns a clear error instead of failing ambiguously.

3) Config safety for non-ONNX binaries

In non-ONNX builds, forcing onnx* config keys now fails fast with a clear message.
Prevents silent misconfiguration when users accidentally pass ONNX-only config into CUDA/OpenCL/Eigen/etc builds.

4) CMake dependency flow

Kept ONNX runtime root wiring via ONNXRUNTIME_ROOT (defaulting to cpp/external/onnxruntime-win-x64-openvino and cpp/external/onnxruntime-linux-x64-openvino).
Added/updated automatic dependency fetch flow for Windows & Linux builds (zlib, onnx, protobuf) through vcpkg when enabled.
ONNX runtime DLLs or SOs are copied to output dir during build on Windows or Linux.

5) Documentation updates

Compiling.md:
- Added explicit Windows & Linux Intel NPU setup steps:
  - Visual Studio Community or VS 2026 Build Tools (Desktop C++)
  - Intel NPU driver install
  - OpenVINO archive install
  - ONNX Runtime build with OpenVINO EP (use_openvino=NPU)
- Added the exact file-copy checklist into cpp/external/onnxruntime-win-x64-openvino.
- Added minimal ONNX backend build command.
README.md:
- Added Intel NPU quick-start section for ONNX/OpenVINO.
- Added minimal commands for:
  - exportonnx (default 19x19)
  - benchmark
  - gtp

Behavior Notes

Multi-device mapping (onnxDeviceToUseThread*) is mainly intended for ONNX providers like CUDA/TensorRT/MIGraphX.
OpenVINO Intel NPU usage is typically single-device.

Validation

ONNX build compiles successfully on Windows & Linuix.
exportonnx works from .bin/.bin.gz -> .onnx.
benchmark/gtp run with onnxProvider=openvino and onnxOpenVINODeviceType=NPU.
Non-ONNX binaries now correctly reject ONNX-only config keys.

Looong01 · 2026-03-16T18:50:44Z

This is screenshot of Sabaki testing:

And the binary release here: https://github.com/Looong01/KataGo-Multi-backends/releases/tag/v1.16.4-openvino

Looong01 · 2026-03-16T18:58:09Z

I partially referenced the code from #1164, and I am very grateful to @ChinChangYang

Looong01 · 2026-04-19T05:12:07Z

This is amazing on my Linux notebook. I am seeing a 3.5x speedup (87.30 vs 25.16 visits/s) compared to OpenCL, which seems unusually slow on my system.

This has been working perfectly for the past month. It would be great to have this feature merged into the official KataGo. Without it, I would have almost had to give up on KataGo after moving to my new PC. Thank you again, @Looong01.

On the Intel Core Ultra 7 255U, OpenCL KataGo is sadly slow, running at less than half the speed of a 5-year-old system with a Core i7-1165G7.

@lightvector

lightvector · 2026-04-20T15:30:55Z

Thanks, I'll also look at this soon.

Looong01 · 2026-04-20T18:56:52Z

Thanks, I'll also look at this soon.

Thanks!

kaorahi · 2026-05-29T10:51:41Z

Thank you for the updates. b37aa25 works fine with the following minor corrections to the ONNX Runtime Backend (Linux) section of Compiling.md.

onnxruntime-win-x64-openvino ==> onnxruntime-linux-x64-openvino
build\Linux\Release ==> build/Linux/Release

In my environment, I also needed to downgrade GCC when running ./build.sh:

CC=gcc-14 CXX=g++-14 ./build.sh ...

At the moment, this is the only branch that runs fast enough for practical use in my environment. I would appreciate official support for this.

foxrainowo · 2026-06-17T09:38:28Z

@Looong01 I don't know if this is a problem with the original or with OpenVino.

Looong01 · 2026-06-17T18:19:17Z

@Looong01 I don't know if this is a problem with the original or with OpenVino.

This is a DEVICE_LOST from the Intel NPU that occurred mid-inference, after roughly 6 hours of self-play (~22,950 games).
The core error:
L0 zeCommandQueueExecuteCommandLists result: ZE_RESULT_ERROR_DEVICE_LOST,
code 0x70000001 – device hung, reset, was removed, or driver update occurred
L0 refers to Level Zero — the OpenVINO intel_npu plugin talks to the NPU through the Level Zero API. The call chain is:
ONNX Runtime → OpenVINO EP (ov_interface.cc:28) → intel_npu plugin (infer_request.cpp:224) → Level Zero (zero_wrappers.cpp) → device lost.
Both errors (subgraph_4 and subgraph_3) have nearly identical timestamps (15:06:27.2041348 and .2041655, ~30 µs apart), which indicates this is not a problem with any individual subgraph — the entire NPU device dropped at that instant, so every subgraph running at the time failed simultaneously.
Why DEVICE_LOST is triggered
0x70000001 is a fairly generic device-level error. The likely causes, ordered by probability for this scenario:

Driver updated mid-run (most likely)

The error message itself says "or driver update occurred." Windows Update silently pushes Intel NPU driver updates in the background. If WU updated the NPU driver during the 6-hour run, the device gets re-enumerated and all existing Level Zero contexts/handles are invalidated, causing in-flight inference to fail with device lost.
2. Long-running resource leak / handle accumulation

This is ~22,950 games with many inferences each — a very large volume. If the intel_npu plugin or this OpenVINO version leaks memory or handles when repeatedly creating/destroying infer requests, accumulation past some threshold can hang the NPU firmware, triggering a GPU-TDR-style reset. The fact that it crashed after 6 hours rather than immediately is consistent with an accumulation-type issue.
3. NPU firmware/driver hang (TDR)

A single inference stalls past the watchdog timeout, the NPU is force-reset, and all subsequent command-queue submissions fail.
4. Thermal/power-induced reset

Possible under sustained load, but NPU power draw is low, so this is the least likely.
Suggested investigation
First, rule out the simplest cause — a driver update:
powershellGet-WinEvent -LogName System | Where-Object {
$_.Message -match "NPU|Intel.*AI Boost|driver"
} | Select-Object TimeCreated, Id, Message -First 20
Focus on whether there were any driver-install / device re-enumeration events around 15:06. You can also check Get-WindowsUpdateLog or the Update history in Settings.
If a driver update is ruled out, other directions:

Add auto-restart + error recovery to the self-play loop. This is the most practical fix: after a device lost, the current process generally has to rebuild the ONNX Runtime session (re-initialize the Level Zero context) — simply catching the exception and continuing will likely fail on all subsequent inferences. The most robust approach is to have an outer script detect this error code, kill the process, and relaunch it, resuming from the last SGF/checkpoint.
Disable automatic driver updates for the NPU to prevent long-running jobs from being interrupted (disable auto-update for the device in Device Manager, or pause Windows Update).
Upgrade OpenVINO / the NPU driver to the latest stable version and re-run, to check whether the leak has been fixed. Reproducing with a short high-frequency stress test is more efficient than blindly running for 6 hours.
If you suspect a leak, monitor NPU memory usage during the run to see whether it grows monotonically.

foxrainowo · 2026-07-20T05:54:31Z

@Looong01
I discovered an issue! I noticed a significant discrepancy between the total amount of visits and the sum of the visits of each candidate move. After selecting a move, its cache size matches the displayed visits amount (with only a very small amount remaining in the cache).

seniorfish · 2026-07-28T13:35:41Z

Looong01 I discovered an issue! I noticed a significant discrepancy between the total amount of visits and the sum of the visits of each candidate move. After selecting a move, its cache size matches the displayed visits amount (with only a very small amount remaining in the cache).

@foxrainowo
From what I’ve observed, this is most likely not a KataGo issue, but a GUI quirk. The LizzieYzy GUI, for the sake of simplicity, hides some low-visit candidate moves by default. If you’d like to see all candidates, go to Settings → Config or press Shift+X to open the config page(综合设置), and set “limit max candidates(限制选点个数)” to 0. That will display every candidate move. And from my own calculations, there’s no discrepancy between the total visit count and the sum of visits across all candidate moves.

foxrainowo · 2026-08-01T17:54:00Z

@seniorfish
Oh, thank you!

Looong01 · 2026-08-05T09:45:47Z

Summary

Adapts the Intel_NPU branch's custom ONNX Runtime backend to the v1.16.5→v1.17.1 upstream sync. The
upstream merge (Merge branch 'master' into Intel_NPU) pulled in TensorRT's new "build via ONNX
graph" path, which added upstream's own cpp/neuralnet/onnxmodelbuilder.cpp — same filename as this
fork's ONNX-Runtime-specific builder, resolved as an add/add conflict in favor of upstream's version.
That silently deleted the fork's builder and changed NeuralNet::createComputeContext's signature
(10 params → 8, ConfigParser& added), leaving USE_BACKEND=ONNX unable to compile.

The rewrite follows the design of the not-yet-merged upstream PR #1222 ("Add ONNX
Runtime backend support") wherever applicable — reusing the same shared OnnxModelBuilder::build()
that TensorRT now uses — while keeping this fork's existing raw .onnx loading and I/O name override
support for non-.bin.gz models.

Changes

cpp/neuralnet/onnxbackend.cpp: full rewrite onto the shared OnnxModelBuilder::build()
architecture (matching PR Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) #1222 / the TensorRT backend). Adds: scale8 FP16-range workaround
(onnxSkipScale8, forced off for -contribute), per-thread OpenVINO device assignment
(onnxOpenVINODeviceTypeThread<N>), per-device-type EP option overrides
(onnxOpenVINODeviceConfig_<NPU|GPU|CPU>_<Option>), onnxOpenVINOPrecision/NumStreams/NumOfThreads/ ModelPriority, onnxTransformerNHWC, KATAGO_DUMP_ONNX debug dump, explicit rejection of the
global useFP16 flag (provider controls precision instead), NeuralNet::setIsWarmup stub. Default
OpenVINO device type kept as NPU (intentional deviation from PR Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) #1222's GPU default, since this
branch targets Intel NPU first). Raw .onnx loading updated to case-insensitive introspection and
PascalCase default I/O names matching the new builder's actual output names.
cpp/neuralnet/onnxmodelbuilder.cpp/.h: cherry-picked PR Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) #1222's two OpenVINO-EP fixes onto
the shared builder (also used by TensorRT): RMSNorm Pow(x,2) instead of Mul(x,x) (OpenVINO's
RMSFusion matcher requires the former, applied to all 4 RMSNorm call sites, not just the 2 PR Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) #1222
touched), and InputSpatial/InputGlobal/[InputMeta]/InputMask declaration order (OpenVINO EP
misroutes the mask tensor with the old InputMask-first order).
cpp/command/misc.cpp: fixed exportonnx, which called the now-removed
OnnxModelBuilder::buildOnnxModel.
cpp/program/setup.cpp: added the onnxSkipScale8 force-off guard for distributed selfplay.
cpp/neuralnet/nneval.h/.cpp, cpp/program/setup.cpp: new generic (backend-agnostic)
per-server-thread max batch size override, nnMaxBatchSizeThread<N>, parallel to the existing
gpuIdxByServerThread. Motivated by community feedback on upstream PR Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) #1222 that NPU+iGPU hybrid
setups need different optimal batch sizes per device (NPU ~1, iGPU larger) in addition to per-thread
device assignment.
cpp/CMakeLists.txt: unified the ONNX backend's protobuf/onnx.proto dependency onto the same
protobuf_generate_cpp mechanism the TensorRT backend uses (removes a redundant separate
onnx-package vcpkg dependency); KATAGO_DEPS_DIR now uses CMAKE_BINARY_DIR instead of
CMAKE_SOURCE_DIR/build (avoids polluting the source tree on out-of-tree builds); Windows vcpkg
triplet default changed to x64-windows-static-md (avoids duplicate abseil_dll.dll instances /
heap corruption).
cpp/configs/gtp_example.cfg, Compiling.md, README.md: documentation refresh for the new
onnx*/nnMaxBatchSizeThread<N> keys and the .bin.gz-direct workflow (no .onnx export needed);
added NPU-only / iGPU-only / NPU+iGPU-hybrid usage examples to the README's ONNX Quick Start
sections (Windows + Linux).

Testing

cmake --build (MSVC, USE_BACKEND=ONNX) succeeds cleanly, no new warnings.
Verified end-to-end on real hardware (Intel NPU + iGPU) with a v1.17 transformer model
(b11c768h12nbt3tflrs-fson-silu.bin.gz):
- NPU-only and iGPU-only benchmark runs complete cleanly with sane outputs.
- Hybrid config (numNNServerThreadsPerModel=2, one thread pinned to NPU, one to iGPU, with
  nnMaxBatchSizeThread0=1 / nnMaxBatchSizeThread1=8) runs both devices concurrently and
  confirms the new per-thread batch size actually takes effect (NPU thread processed rows 1:1 with
  batches, iGPU thread batched multiple rows per call).
- Confirmed the OpenVINO-EP input declaration order fix and RMSNorm fix are exercised for real
  (transformerNHWC=true, transformer trunk model) without shape-mismatch errors.

Looong01 · 2026-08-05T10:56:53Z

Intel GPU test on Ultra 9 275HX:

(base) PS C:\Envs\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64> .\katago.exe benchmark -model b11c768h12nbt3tflrs-fson-silu.bin.gz -config gtp_npu.cfg -override-config "onnxOpenVINODeviceType=GPU"
2026-08-05 17:27:45+0800: Running with following config:
allowResignation = true
lagBuffer = 1.0
logAllGTPCommunication = true
logDir = gtp_logs
logSearchInfo = true
logSearchInfoForChosenMove = false
logToStderr = false
maxTimePondering = 60.0
maxVisits = 500
numSearchThreads = 1
onnxOpenVINODeviceType = GPU
onnxProvider = openvino
ponderingEnabled = false
resignConsecTurns = 3
resignThreshold = -0.90
rules = tromp-taylor
searchFactorAfterOnePass = 0.50
searchFactorAfterTwoPass = 0.25
searchFactorWhenWinning = 0.40
searchFactorWhenWinningThreshold = 0.95

2026-08-05 17:27:45+0800: Loading model and initializing benchmark...
2026-08-05 17:27:45+0800: Testing with default positions for board size: 19
2026-08-05 17:27:45+0800: nnRandSeed0 = 16002040592701319046
2026-08-05 17:27:45+0800: After dedups: nnModelFile0 = b11c768h12nbt3tflrs-fson-silu.bin.gz useFP16 auto
2026-08-05 17:27:45+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 17:27:47+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 17:27:47+0800: ONNX backend thread 0: Model version 17
2026-08-05 17:27:47+0800: ONNX backend thread 0: Model name: b11c768h12nbt3tflrs-fson-silu
2026-08-05 17:27:47+0800: ONNX backend thread 0: provider=openvino deviceIdx=GPU
2026-08-05 17:27:47+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 17:27:47+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 17:27:48+0800: ONNX backend: ONNX graph built (355300041 bytes)
2026-08-05 17:27:48+0800: ONNX backend: creating session...
2026-08-05 17:27:48+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=GPU
2026-08-05 17:27:56+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 17:27:56+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 17:27:56+0800: ONNX backend: session created, inputs=3 outputs=5

2026-08-05 17:28:12+0800: Loaded config gtp_npu.cfg and/or command-line and query overrides
2026-08-05 17:28:12+0800: Loaded model b11c768h12nbt3tflrs-fson-silu.bin.gz

Testing using 800 visits.
  If you have a good GPU, you might increase this using "-visits N" to get more accurate results.
  If you have a weak GPU and this is taking forever, you can decrease it instead to finish the benchmark faster.

You are currently using the ONNX Runtime version of KataGo.
Your GTP config is currently set to onnxProvider = openvino
OpenVINO device type = GPU
For Intel NPU, typically set onnxOpenVINODeviceType = NPU.
OpenVINO/NPU usually uses a single device; onnxDeviceToUseThread* is typically for cuda/trt/migraphx providers.

Your GTP config is currently set to use numSearchThreads = 1
Automatically trying different numbers of threads to home in on the best (board size 19x19):

2026-08-05 17:28:12+0800: GPU -1 finishing, processed 5 rows 5 batches
2026-08-05 17:28:12+0800: nnRandSeed0 = 15077079755957294086
2026-08-05 17:28:12+0800: After dedups: nnModelFile0 = b11c768h12nbt3tflrs-fson-silu.bin.gz useFP16 auto
2026-08-05 17:28:12+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 17:28:14+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 17:28:14+0800: ONNX backend thread 0: Model version 17
2026-08-05 17:28:14+0800: ONNX backend thread 0: Model name: b11c768h12nbt3tflrs-fson-silu
2026-08-05 17:28:14+0800: ONNX backend thread 0: provider=openvino deviceIdx=GPU
2026-08-05 17:28:14+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 17:28:14+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 17:28:15+0800: ONNX backend: ONNX graph built (355300041 bytes)
2026-08-05 17:28:15+0800: ONNX backend: creating session...
2026-08-05 17:28:15+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=GPU
2026-08-05 17:28:19+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 17:28:19+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 17:28:19+0800: ONNX backend: session created, inputs=3 outputs=5


Possible numbers of threads to test: 1, 2, 3, 4, 5, 6, 8, 10, 12, 16, 20, 24, 32,

numSearchThreads =  5: 10 / 10 positions, visits/s = 9.99 nnEvals/s = 9.99 nnBatches/s = 4.01 avgBatchSize = 2.49 (804.3 secs)
numSearchThreads = 12: 10 / 10 positions, visits/s = 11.90 nnEvals/s = 11.10 nnBatches/s = 1.85 avgBatchSize = 6.01 (681.3 secs)
numSearchThreads =  3: 10 / 10 positions, visits/s = 9.37 nnEvals/s = 9.37 nnBatches/s = 6.25 avgBatchSize = 1.50 (856.0 secs)
numSearchThreads =  6: 10 / 10 positions, visits/s = 10.35 nnEvals/s = 10.31 nnBatches/s = 3.45 avgBatchSize = 2.99 (777.4 secs)
numSearchThreads =  8: 10 / 10 positions, visits/s = 11.09 nnEvals/s = 10.71 nnBatches/s = 2.69 avgBatchSize = 3.99 (727.6 secs)
numSearchThreads = 10: 10 / 10 positions, visits/s = 11.63 nnEvals/s = 11.26 nnBatches/s = 2.26 avgBatchSize = 4.98 (695.2 secs)


Ordered summary of results:

numSearchThreads =  3: 10 / 10 positions, visits/s = 9.37 nnEvals/s = 9.37 nnBatches/s = 6.25 avgBatchSize = 1.50 (856.0 secs) (EloDiff baseline)
numSearchThreads =  5: 10 / 10 positions, visits/s = 9.99 nnEvals/s = 9.99 nnBatches/s = 4.01 avgBatchSize = 2.49 (804.3 secs) (EloDiff -1)
numSearchThreads =  6: 10 / 10 positions, visits/s = 10.35 nnEvals/s = 10.31 nnBatches/s = 3.45 avgBatchSize = 2.99 (777.4 secs) (EloDiff +0)
numSearchThreads =  8: 10 / 10 positions, visits/s = 11.09 nnEvals/s = 10.71 nnBatches/s = 2.69 avgBatchSize = 3.99 (727.6 secs) (EloDiff +1)
numSearchThreads = 10: 10 / 10 positions, visits/s = 11.63 nnEvals/s = 11.26 nnBatches/s = 2.26 avgBatchSize = 4.98 (695.2 secs) (EloDiff -5)
numSearchThreads = 12: 10 / 10 positions, visits/s = 11.90 nnEvals/s = 11.10 nnBatches/s = 1.85 avgBatchSize = 6.01 (681.3 secs) (EloDiff -20)


Based on some test data, each speed doubling gains perhaps ~250 Elo by searching deeper.
Based on some test data, each thread costs perhaps 7 Elo if using 800 visits, and 2 Elo if using 5000 visits (by making MCTS worse).
So APPROXIMATELY based on this benchmark, if you intend to do a 5 second search:
numSearchThreads =  3: (baseline)
numSearchThreads =  5:    -1 Elo
numSearchThreads =  6:    +0 Elo
numSearchThreads =  8:    +1 Elo (recommended)
numSearchThreads = 10:    -5 Elo
numSearchThreads = 12:   -20 Elo

If you care about performance, you may want to edit numSearchThreads in gtp_npu.cfg and/or command-line and query overrides based on the above results!
If you intend to do much longer searches, configure the seconds per game move you expect with the '-time' flag and benchmark again.
If you intend to do short or fixed-visit searches, use lower numSearchThreads for better strength, high threads will weaken strength.
If interested see also other notes about performance and mem usage in the top of gtp_npu.cfg and/or command-line and query overrides

2026-08-05 18:44:05+0800: GPU -1 finishing, processed 47255 rows 16046 batches

foxrainowo · 2026-08-05T13:02:24Z

I'm having issues using gtp_npu.cfg. It stays at visits/s = -nan(ind) for about ten minutes, then reports an error and exits.

PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\katago.exe benchmark -model "C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz" -config "C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_npu.cfg"
2026-08-05 19:56:55+0800: Running with following config:
allowResignation = true
lagBuffer = 1.0
logAllGTPCommunication = true
logDir = gtp_logs
logSearchInfo = true
logSearchInfoForChosenMove = false
logToStderr = false
maxTimePondering = 60.0
maxVisits = 500
numSearchThreads = 1
onnxOpenVINODeviceType = NPU
onnxProvider = openvino
ponderingEnabled = false
resignConsecTurns = 3
resignThreshold = -0.90
rules = tromp-taylor
searchFactorAfterOnePass = 0.50
searchFactorAfterTwoPass = 0.25
searchFactorWhenWinning = 0.40
searchFactorWhenWinningThreshold = 0.95

2026-08-05 19:56:55+0800: Loading model and initializing benchmark...
2026-08-05 19:56:55+0800: Testing with default positions for board size: 19
2026-08-05 19:56:55+0800: nnRandSeed0 = 6661592873850738289
2026-08-05 19:56:55+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz useFP16 auto
2026-08-05 19:56:55+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 19:57:04+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 19:57:04+0800: ONNX backend thread 0: Model version 15
2026-08-05 19:57:04+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-fdx6d-s11272M-d5935M
2026-08-05 19:57:04+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-05 19:57:04+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 19:57:04+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 19:57:05+0800: ONNX backend: ONNX graph built (930658630 bytes)
2026-08-05 19:57:05+0800: ONNX backend: creating session...
2026-08-05 19:57:05+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=NPU
2026-08-05 19:57:08+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 19:57:08+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 19:57:08+0800: ONNX backend: session created, inputs=3 outputs=5

2026-08-05 19:57:11+0800: Loaded config C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_npu.cfg
2026-08-05 19:57:11+0800: Loaded model C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz

Testing using 800 visits.
  If you have a good GPU, you might increase this using "-visits N" to get more accurate results.
  If you have a weak GPU and this is taking forever, you can decrease it instead to finish the benchmark faster.

You are currently using the ONNX Runtime version of KataGo.
Your GTP config is currently set to onnxProvider = openvino
OpenVINO device type = NPU
For Intel NPU, typically set onnxOpenVINODeviceType = NPU.
OpenVINO/NPU usually uses a single device; onnxDeviceToUseThread* is typically for cuda/trt/migraphx providers.

Your GTP config is currently set to use numSearchThreads = 1
Automatically trying different numbers of threads to home in on the best (board size 19x19):

2026-08-05 19:57:11+0800: GPU -1 finishing, processed 5 rows 5 batches
2026-08-05 19:57:11+0800: nnRandSeed0 = 9916478682849984213
2026-08-05 19:57:11+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz useFP16 auto
2026-08-05 19:57:11+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 19:57:20+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 19:57:20+0800: ONNX backend thread 0: Model version 15
2026-08-05 19:57:20+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-fdx6d-s11272M-d5935M
2026-08-05 19:57:20+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-05 19:57:20+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 19:57:20+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 19:57:21+0800: ONNX backend: ONNX graph built (930658630 bytes)
2026-08-05 19:57:22+0800: ONNX backend: creating session...
2026-08-05 19:57:22+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=NPU
2026-08-05 19:57:24+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 19:57:24+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 19:57:24+0800: ONNX backend: session created, inputs=3 outputs=5


Possible numbers of threads to test: 1, 2, 3, 4, 5, 6, 8, 10, 12, 16, 20, 24, 32,

numSearchThreads =  5: 1 / 10 positions, visits/s = 1.31 (616.0 secs)      2026-08-05 20:07:58.5189468 [E:onnxruntime:, sequential_executor.cc:671 onnxruntime::ExecuteKernel] Non-zero status code returned while running OpenVINO-EP-subgraph_1 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0' Status Message: C:\Users\Loong\Codes\onnxruntime\onnxruntime\core\providers\openvino\ov_interface.cc:28 void __cdecl onnxruntime::openvino_ep::OvExceptionBoundary<false,class onnxruntime::openvino_ep::OVInferRequest::Infer::<lambda_1>,>(class onnxruntime::openvino_ep::OVInferRequest::Infer::<lambda_1> &&,struct std::basic_format_string<char> &&) [OpenVINO-EP] In Error Couldn't start Inference: Exception from src\inference\src\cpp\infer_request.cpp:224:
Exception from src\plugins\intel_npu\src\utils\src\zero\zero_wrappers.cpp:354:
L0 zeCommandQueueExecuteCommandLists result: ZE_RESULT_ERROR_DEVICE_LOST, code 0x70000001 - device hung, reset, was removed, or driver update occurred

PS C:\Users\foxrain>

Although gtp_npu_gpu_hybrid and gtp_gpu.cfg work normally, I found that their performance is gtp_gpu.cfg >= gtp_npu_gpu_hybrid, meaning the hybrid configuration is actually slower, so it's better to just use the GPU.
Additionally, it did not generate the compilation cache at the specified location in gtp. But it will generate in match.

PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\katago.exe benchmark -model "C:\Users\foxrain\Downloads\b11c768h12nbt3tflrs-fson-silu.bin.gz" -config "C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_npu_gpu_hybrid.cfg"
2026-08-05 18:24:33+0800: Running with following config:
allowResignation = true
lagBuffer = 1.0
logAllGTPCommunication = true
logDir = gtp_logs
logSearchInfo = true
logSearchInfoForChosenMove = false
logToStderr = false
maxTimePondering = 60.0
maxVisits = 500
nnMaxBatchSizeThread0 = 1
nnMaxBatchSizeThread1 = 8
numNNServerThreadsPerModel = 2
numSearchThreads = 1
onnxOpenVINODeviceType = NPU
onnxOpenVINODeviceTypeThread0 = NPU
onnxOpenVINODeviceTypeThread1 = GPU
onnxProvider = openvino
ponderingEnabled = false
resignConsecTurns = 3
resignThreshold = -0.90
rules = japanese
searchFactorAfterOnePass = 0.50
searchFactorAfterTwoPass = 0.25
searchFactorWhenWinning = 0.40
searchFactorWhenWinningThreshold = 0.95

2026-08-05 18:24:33+0800: Loading model and initializing benchmark...
2026-08-05 18:24:33+0800: Testing with default positions for board size: 19
2026-08-05 18:24:33+0800: nnRandSeed0 = 6992435971789407022
2026-08-05 18:24:33+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\b11c768h12nbt3tflrs-fson-silu.bin.gz useFP16 auto
2026-08-05 18:24:33+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 18:24:36+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 18:24:36+0800: ONNX backend thread 0: Model version 17
2026-08-05 18:24:36+0800: ONNX backend thread 0: Model name: b11c768h12nbt3tflrs-fson-silu
2026-08-05 18:24:36+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-05 18:24:36+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 18:24:36+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 18:24:36+0800: ONNX backend thread 1: Model version 17
2026-08-05 18:24:36+0800: ONNX backend thread 1: Model name: b11c768h12nbt3tflrs-fson-silu
2026-08-05 18:24:36+0800: ONNX backend thread 1: provider=openvino deviceIdx=GPU
2026-08-05 18:24:36+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 18:24:36+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 18:24:37+0800: ONNX backend: ONNX graph built (355300041 bytes)
2026-08-05 18:24:37+0800: ONNX backend: ONNX graph built (355300041 bytes)
2026-08-05 18:24:37+0800: ONNX backend: creating session...
2026-08-05 18:24:37+0800: ONNX backend: creating session...
2026-08-05 18:24:40+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=NPU
2026-08-05 18:24:40+0800: ONNX backend: OpenVINO EP enabled for thread 1, device_type=GPU
2026-08-05 18:24:41+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 18:24:41+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 18:24:41+0800: ONNX backend: session created, inputs=3 outputs=5
2026-08-05 18:24:45+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 18:24:45+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 18:24:45+0800: ONNX backend: session created, inputs=3 outputs=5

2026-08-05 18:24:48+0800: Loaded config C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_npu_gpu_hybrid.cfg
2026-08-05 18:24:48+0800: Loaded model C:\Users\foxrain\Downloads\b11c768h12nbt3tflrs-fson-silu.bin.gz

Testing using 800 visits.
  If you have a good GPU, you might increase this using "-visits N" to get more accurate results.
  If you have a weak GPU and this is taking forever, you can decrease it instead to finish the benchmark faster.

You are currently using the ONNX Runtime version of KataGo.
Your GTP config is currently set to onnxProvider = openvino
OpenVINO device type = NPU
For Intel NPU, typically set onnxOpenVINODeviceType = NPU.
OpenVINO/NPU usually uses a single device; onnxDeviceToUseThread* is typically for cuda/trt/migraphx providers.

Your GTP config is currently set to use numSearchThreads = 1
Automatically trying different numbers of threads to home in on the best (board size 19x19):

2026-08-05 18:24:48+0800: GPU -1 finishing, processed 1 rows 1 batches
2026-08-05 18:24:48+0800: GPU -1 finishing, processed 4 rows 4 batches
2026-08-05 18:24:48+0800: nnRandSeed0 = 135529463997515913
2026-08-05 18:24:48+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\b11c768h12nbt3tflrs-fson-silu.bin.gz useFP16 auto
2026-08-05 18:24:48+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 18:24:51+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 18:24:51+0800: ONNX backend thread 0: Model version 17
2026-08-05 18:24:51+0800: ONNX backend thread 0: Model name: b11c768h12nbt3tflrs-fson-silu
2026-08-05 18:24:51+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-05 18:24:51+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 18:24:51+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 18:24:51+0800: ONNX backend thread 1: Model version 17
2026-08-05 18:24:51+0800: ONNX backend thread 1: Model name: b11c768h12nbt3tflrs-fson-silu
2026-08-05 18:24:51+0800: ONNX backend thread 1: provider=openvino deviceIdx=GPU
2026-08-05 18:24:51+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 18:24:51+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 18:24:51+0800: ONNX backend: ONNX graph built (355300041 bytes)
2026-08-05 18:24:51+0800: ONNX backend: ONNX graph built (355300041 bytes)
2026-08-05 18:24:51+0800: ONNX backend: creating session...
2026-08-05 18:24:51+0800: ONNX backend: creating session...
2026-08-05 18:24:51+0800: ONNX backend: OpenVINO EP enabled for thread 1, device_type=GPU
2026-08-05 18:24:51+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=NPU
2026-08-05 18:24:52+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 18:24:52+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 18:24:52+0800: ONNX backend: session created, inputs=3 outputs=5
2026-08-05 18:24:55+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 18:24:55+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 18:24:55+0800: ONNX backend: session created, inputs=3 outputs=5


Possible numbers of threads to test: 1, 2, 3, 4, 5, 6, 8, 10, 12, 16, 20, 24, 32,

numSearchThreads =  5: 10 / 10 positions, visits/s = 51.07 nnEvals/s = 44.07 nnBatches/s = 23.10 avgBatchSize = 1.91 (157.4 secs)
numSearchThreads = 12: 10 / 10 positions, visits/s = 66.09 nnEvals/s = 57.16 nnBatches/s = 11.08 avgBatchSize = 5.16 (122.7 secs)
numSearchThreads = 10: 10 / 10 positions, visits/s = 68.09 nnEvals/s = 58.36 nnBatches/s = 12.06 avgBatchSize = 4.84 (118.8 secs)
numSearchThreads = 20: 10 / 10 positions, visits/s = 72.20 nnEvals/s = 64.16 nnBatches/s = 11.81 avgBatchSize = 5.43 (113.4 secs)
numSearchThreads =  8: 10 / 10 positions, visits/s = 65.25 nnEvals/s = 54.81 nnBatches/s = 13.95 avgBatchSize = 3.93 (123.7 secs)
numSearchThreads =  6: 10 / 10 positions, visits/s = 61.63 nnEvals/s = 52.07 nnBatches/s = 18.39 avgBatchSize = 2.83 (130.6 secs)


Ordered summary of results:

numSearchThreads =  5: 10 / 10 positions, visits/s = 51.07 nnEvals/s = 44.07 nnBatches/s = 23.10 avgBatchSize = 1.91 (157.4 secs) (EloDiff baseline)
numSearchThreads =  6: 10 / 10 positions, visits/s = 61.63 nnEvals/s = 52.07 nnBatches/s = 18.39 avgBatchSize = 2.83 (130.6 secs) (EloDiff +60)
numSearchThreads =  8: 10 / 10 positions, visits/s = 65.25 nnEvals/s = 54.81 nnBatches/s = 13.95 avgBatchSize = 3.93 (123.7 secs) (EloDiff +63)
numSearchThreads = 10: 10 / 10 positions, visits/s = 68.09 nnEvals/s = 58.36 nnBatches/s = 12.06 avgBatchSize = 4.84 (118.8 secs) (EloDiff +60)
numSearchThreads = 12: 10 / 10 positions, visits/s = 66.09 nnEvals/s = 57.16 nnBatches/s = 11.08 avgBatchSize = 5.16 (122.7 secs) (EloDiff +30)
numSearchThreads = 20: 10 / 10 positions, visits/s = 72.20 nnEvals/s = 64.16 nnBatches/s = 11.81 avgBatchSize = 5.43 (113.4 secs) (EloDiff -9)


Based on some test data, each speed doubling gains perhaps ~250 Elo by searching deeper.
Based on some test data, each thread costs perhaps 7 Elo if using 800 visits, and 2 Elo if using 5000 visits (by making MCTS worse).
So APPROXIMATELY based on this benchmark, if you intend to do a 5 second search:
numSearchThreads =  5: (baseline)
numSearchThreads =  6:   +60 Elo
numSearchThreads =  8:   +63 Elo (recommended)
numSearchThreads = 10:   +60 Elo
numSearchThreads = 12:   +30 Elo
numSearchThreads = 20:    -9 Elo

If you care about performance, you may want to edit numSearchThreads in C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_npu_gpu_hybrid.cfg based on the above results!
If you intend to do much longer searches, configure the seconds per game move you expect with the '-time' flag and benchmark again.
If you intend to do short or fixed-visit searches, use lower numSearchThreads for better strength, high threads will weaken strength.
If interested see also other notes about performance and mem usage in the top of C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_npu_gpu_hybrid.cfg

2026-08-05 18:37:48+0800: GPU -1 finishing, processed 38472 rows 8626 batches
2026-08-05 18:37:48+0800: GPU -1 finishing, processed 3276 rows 3276 batches

PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\katago.exe benchmark -model "C:\Users\foxrain\Downloads\b11c768h12nbt3tflrs-fson-silu.bin.gz" -config "C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_gpu.cfg"
2026-08-05 20:46:55+0800: Running with following config:
allowResignation = true
lagBuffer = 1.0
logAllGTPCommunication = true
logDir = gtp_logs
logSearchInfo = true
logSearchInfoForChosenMove = false
logToStderr = false
maxTimePondering = 60.0
maxVisits = 500
numSearchThreads = 4
onnxOpenVINODeviceType = GPU
onnxProvider = openvino
ponderingEnabled = false
resignConsecTurns = 3
resignThreshold = -0.90
rules = tromp-taylor
searchFactorAfterOnePass = 0.50
searchFactorAfterTwoPass = 0.25
searchFactorWhenWinning = 0.40
searchFactorWhenWinningThreshold = 0.95

2026-08-05 20:46:55+0800: Loading model and initializing benchmark...
2026-08-05 20:46:55+0800: Testing with default positions for board size: 19
2026-08-05 20:46:55+0800: nnRandSeed0 = 7983874965468556636
2026-08-05 20:46:55+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\b11c768h12nbt3tflrs-fson-silu.bin.gz useFP16 auto
2026-08-05 20:46:55+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 20:46:57+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 20:46:57+0800: ONNX backend thread 0: Model version 17
2026-08-05 20:46:57+0800: ONNX backend thread 0: Model name: b11c768h12nbt3tflrs-fson-silu
2026-08-05 20:46:57+0800: ONNX backend thread 0: provider=openvino deviceIdx=GPU
2026-08-05 20:46:57+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 20:46:57+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 20:46:58+0800: ONNX backend: ONNX graph built (355300041 bytes)
2026-08-05 20:46:58+0800: ONNX backend: creating session...
2026-08-05 20:46:58+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=GPU
2026-08-05 20:47:01+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 20:47:01+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 20:47:01+0800: ONNX backend: session created, inputs=3 outputs=5

2026-08-05 20:47:02+0800: Loaded config C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_gpu.cfg
2026-08-05 20:47:02+0800: Loaded model C:\Users\foxrain\Downloads\b11c768h12nbt3tflrs-fson-silu.bin.gz

Testing using 800 visits.
  If you have a good GPU, you might increase this using "-visits N" to get more accurate results.
  If you have a weak GPU and this is taking forever, you can decrease it instead to finish the benchmark faster.

You are currently using the ONNX Runtime version of KataGo.
Your GTP config is currently set to onnxProvider = openvino
OpenVINO device type = GPU
For Intel NPU, typically set onnxOpenVINODeviceType = NPU.
OpenVINO/NPU usually uses a single device; onnxDeviceToUseThread* is typically for cuda/trt/migraphx providers.

Your GTP config is currently set to use numSearchThreads = 4
Automatically trying different numbers of threads to home in on the best (board size 19x19):

2026-08-05 20:47:02+0800: GPU -1 finishing, processed 5 rows 5 batches
2026-08-05 20:47:02+0800: nnRandSeed0 = 12156296877601320369
2026-08-05 20:47:02+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\b11c768h12nbt3tflrs-fson-silu.bin.gz useFP16 auto
2026-08-05 20:47:02+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 20:47:09+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 20:47:09+0800: ONNX backend thread 0: Model version 17
2026-08-05 20:47:09+0800: ONNX backend thread 0: Model name: b11c768h12nbt3tflrs-fson-silu
2026-08-05 20:47:09+0800: ONNX backend thread 0: provider=openvino deviceIdx=GPU
2026-08-05 20:47:09+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 20:47:09+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 20:47:10+0800: ONNX backend: ONNX graph built (355300041 bytes)
2026-08-05 20:47:10+0800: ONNX backend: creating session...
2026-08-05 20:47:10+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=GPU
2026-08-05 20:47:13+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 20:47:13+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 20:47:13+0800: ONNX backend: session created, inputs=3 outputs=5


Possible numbers of threads to test: 1, 2, 3, 4, 5, 6, 8, 10, 12, 16, 20, 24, 32,

numSearchThreads =  5: 10 / 10 positions, visits/s = 80.77 nnEvals/s = 68.57 nnBatches/s = 17.55 avgBatchSize = 3.91 (99.5 secs)
numSearchThreads = 12: 10 / 10 positions, visits/s = 75.99 nnEvals/s = 65.56 nnBatches/s = 6.15 avgBatchSize = 10.66 (106.7 secs)
numSearchThreads =  3: 10 / 10 positions, visits/s = 61.40 nnEvals/s = 51.35 nnBatches/s = 25.86 avgBatchSize = 1.99 (130.6 secs)
numSearchThreads =  6: 10 / 10 positions, visits/s = 65.82 nnEvals/s = 56.23 nnBatches/s = 11.98 avgBatchSize = 4.69 (122.3 secs)
numSearchThreads =  2: 10 / 10 positions, visits/s = 29.03 nnEvals/s = 24.63 nnBatches/s = 22.49 avgBatchSize = 1.10 (276.0 secs)
numSearchThreads =  4: 10 / 10 positions, visits/s = 64.18 nnEvals/s = 54.39 nnBatches/s = 19.21 avgBatchSize = 2.83 (125.1 secs)


Ordered summary of results:

numSearchThreads =  2: 10 / 10 positions, visits/s = 29.03 nnEvals/s = 24.63 nnBatches/s = 22.49 avgBatchSize = 1.10 (276.0 secs) (EloDiff baseline)
numSearchThreads =  3: 10 / 10 positions, visits/s = 61.40 nnEvals/s = 51.35 nnBatches/s = 25.86 avgBatchSize = 1.99 (130.6 secs) (EloDiff +263)
numSearchThreads =  4: 10 / 10 positions, visits/s = 64.18 nnEvals/s = 54.39 nnBatches/s = 19.21 avgBatchSize = 2.83 (125.1 secs) (EloDiff +270)
numSearchThreads =  5: 10 / 10 positions, visits/s = 80.77 nnEvals/s = 68.57 nnBatches/s = 17.55 avgBatchSize = 3.91 (99.5 secs) (EloDiff +346)
numSearchThreads =  6: 10 / 10 positions, visits/s = 65.82 nnEvals/s = 56.23 nnBatches/s = 11.98 avgBatchSize = 4.69 (122.3 secs) (EloDiff +261)
numSearchThreads = 12: 10 / 10 positions, visits/s = 75.99 nnEvals/s = 65.56 nnBatches/s = 6.15 avgBatchSize = 10.66 (106.7 secs) (EloDiff +260)


Based on some test data, each speed doubling gains perhaps ~250 Elo by searching deeper.
Based on some test data, each thread costs perhaps 7 Elo if using 800 visits, and 2 Elo if using 5000 visits (by making MCTS worse).
So APPROXIMATELY based on this benchmark, if you intend to do a 5 second search:
numSearchThreads =  2: (baseline)
numSearchThreads =  3:  +263 Elo
numSearchThreads =  4:  +270 Elo
numSearchThreads =  5:  +346 Elo (recommended)
numSearchThreads =  6:  +261 Elo
numSearchThreads = 12:  +260 Elo

If you care about performance, you may want to edit numSearchThreads in C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_gpu.cfg based on the above results!
If you intend to do much longer searches, configure the seconds per game move you expect with the '-time' flag and benchmark again.
If you intend to do short or fixed-visit searches, use lower numSearchThreads for better strength, high threads will weaken strength.
If interested see also other notes about performance and mem usage in the top of C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_gpu.cfg

2026-08-05 21:01:36+0800: GPU -1 finishing, processed 41013 rows 15861 batches

PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\katago.exe benchmark -model "C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz" -config "C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_npu_gpu_hybrid.cfg"
2026-08-05 18:46:11+0800: Running with following config:
allowResignation = true
lagBuffer = 1.0
logAllGTPCommunication = true
logDir = gtp_logs
logSearchInfo = true
logSearchInfoForChosenMove = false
logToStderr = false
maxTimePondering = 60.0
maxVisits = 500
nnMaxBatchSizeThread0 = 1
nnMaxBatchSizeThread1 = 8
numNNServerThreadsPerModel = 2
numSearchThreads = 6
onnxOpenVINODeviceType = NPU
onnxOpenVINODeviceTypeThread0 = NPU
onnxOpenVINODeviceTypeThread1 = GPU
onnxProvider = openvino
ponderingEnabled = false
resignConsecTurns = 3
resignThreshold = -0.90
rules = japanese
searchFactorAfterOnePass = 0.50
searchFactorAfterTwoPass = 0.25
searchFactorWhenWinning = 0.40
searchFactorWhenWinningThreshold = 0.95

2026-08-05 18:46:11+0800: Loading model and initializing benchmark...
2026-08-05 18:46:11+0800: Testing with default positions for board size: 19
2026-08-05 18:46:11+0800: nnRandSeed0 = 10456369991387597996
2026-08-05 18:46:11+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz useFP16 auto
2026-08-05 18:46:11+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 18:46:40+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 18:46:40+0800: ONNX backend thread 0: Model version 15
2026-08-05 18:46:40+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-fdx6d-s11272M-d5935M
2026-08-05 18:46:40+0800: ONNX backend thread 1: Model version 15
2026-08-05 18:46:40+0800: ONNX backend thread 1: Model name: kata1-zhizi-b40c768nbt-fdx6d-s11272M-d5935M
2026-08-05 18:46:40+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-05 18:46:40+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 18:46:40+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 18:46:40+0800: ONNX backend thread 1: provider=openvino deviceIdx=GPU
2026-08-05 18:46:40+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 18:46:40+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 18:46:42+0800: ONNX backend: ONNX graph built (930658630 bytes)
2026-08-05 18:46:42+0800: ONNX backend: ONNX graph built (930658630 bytes)
2026-08-05 18:46:42+0800: ONNX backend: creating session...
2026-08-05 18:46:42+0800: ONNX backend: creating session...
2026-08-05 18:46:42+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=NPU
2026-08-05 18:46:42+0800: ONNX backend: OpenVINO EP enabled for thread 1, device_type=GPU
2026-08-05 18:46:44+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 18:46:44+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 18:46:44+0800: ONNX backend: session created, inputs=3 outputs=5
2026-08-05 18:46:47+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 18:46:47+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 18:46:47+0800: ONNX backend: session created, inputs=3 outputs=5

2026-08-05 18:46:57+0800: Loaded config C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_npu_gpu_hybrid.cfg
2026-08-05 18:46:57+0800: Loaded model C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz

Testing using 800 visits.
  If you have a good GPU, you might increase this using "-visits N" to get more accurate results.
  If you have a weak GPU and this is taking forever, you can decrease it instead to finish the benchmark faster.

You are currently using the ONNX Runtime version of KataGo.
Your GTP config is currently set to onnxProvider = openvino
OpenVINO device type = NPU
For Intel NPU, typically set onnxOpenVINODeviceType = NPU.
OpenVINO/NPU usually uses a single device; onnxDeviceToUseThread* is typically for cuda/trt/migraphx providers.

Your GTP config is currently set to use numSearchThreads = 6
Automatically trying different numbers of threads to home in on the best (board size 19x19):

2026-08-05 18:46:57+0800: GPU -1 finishing, processed 2 rows 2 batches
2026-08-05 18:46:57+0800: GPU -1 finishing, processed 3 rows 3 batches
2026-08-05 18:46:57+0800: nnRandSeed0 = 3859048696684069015
2026-08-05 18:46:57+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz useFP16 auto
2026-08-05 18:46:57+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 18:47:15+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 18:47:15+0800: ONNX backend thread 1: Model version 15
2026-08-05 18:47:15+0800: ONNX backend thread 1: Model name: kata1-zhizi-b40c768nbt-fdx6d-s11272M-d5935M
2026-08-05 18:47:15+0800: ONNX backend thread 1: provider=openvino deviceIdx=GPU
2026-08-05 18:47:15+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 18:47:15+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 18:47:15+0800: ONNX backend thread 0: Model version 15
2026-08-05 18:47:15+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-fdx6d-s11272M-d5935M
2026-08-05 18:47:15+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-05 18:47:15+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 18:47:15+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 18:47:16+0800: ONNX backend: ONNX graph built (930658630 bytes)
2026-08-05 18:47:16+0800: ONNX backend: ONNX graph built (930658630 bytes)
2026-08-05 18:47:16+0800: ONNX backend: creating session...
2026-08-05 18:47:16+0800: ONNX backend: OpenVINO EP enabled for thread 1, device_type=GPU
2026-08-05 18:47:16+0800: ONNX backend: creating session...
2026-08-05 18:47:16+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=NPU
2026-08-05 18:47:19+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 18:47:19+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 18:47:19+0800: ONNX backend: session created, inputs=3 outputs=5
2026-08-05 18:47:22+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 18:47:22+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 18:47:22+0800: ONNX backend: session created, inputs=3 outputs=5


Possible numbers of threads to test: 1, 2, 3, 4, 5, 6, 8, 10, 12, 16, 20, 24, 32,

numSearchThreads =  5: 10 / 10 positions, visits/s = 23.42 nnEvals/s = 22.33 nnBatches/s = 11.58 avgBatchSize = 1.93 (343.2 secs)
numSearchThreads = 12: 10 / 10 positions, visits/s = 21.99 nnEvals/s = 21.67 nnBatches/s = 7.41 avgBatchSize = 2.92 (368.8 secs)
numSearchThreads =  3: 10 / 10 positions, visits/s = 22.52 nnEvals/s = 20.84 nnBatches/s = 20.43 avgBatchSize = 1.02 (356.2 secs)
numSearchThreads =  6: 10 / 10 positions, visits/s = 26.09 nnEvals/s = 24.67 nnBatches/s = 10.16 avgBatchSize = 2.43 (308.5 secs)
numSearchThreads =  8: 10 / 10 positions, visits/s = 26.15 nnEvals/s = 25.83 nnBatches/s = 8.72 avgBatchSize = 2.96 (308.6 secs)
numSearchThreads = 10: 10 / 10 positions, visits/s = 25.63 nnEvals/s = 25.14 nnBatches/s = 7.77 avgBatchSize = 3.23 (315.6 secs)


Ordered summary of results:

numSearchThreads =  3: 10 / 10 positions, visits/s = 22.52 nnEvals/s = 20.84 nnBatches/s = 20.43 avgBatchSize = 1.02 (356.2 secs) (EloDiff baseline)
numSearchThreads =  5: 10 / 10 positions, visits/s = 23.42 nnEvals/s = 22.33 nnBatches/s = 11.58 avgBatchSize = 1.93 (343.2 secs) (EloDiff -8)
numSearchThreads =  6: 10 / 10 positions, visits/s = 26.09 nnEvals/s = 24.67 nnBatches/s = 10.16 avgBatchSize = 2.43 (308.5 secs) (EloDiff +20)
numSearchThreads =  8: 10 / 10 positions, visits/s = 26.15 nnEvals/s = 25.83 nnBatches/s = 8.72 avgBatchSize = 2.96 (308.6 secs) (EloDiff -1)
numSearchThreads = 10: 10 / 10 positions, visits/s = 25.63 nnEvals/s = 25.14 nnBatches/s = 7.77 avgBatchSize = 3.23 (315.6 secs) (EloDiff -31)
numSearchThreads = 12: 10 / 10 positions, visits/s = 21.99 nnEvals/s = 21.67 nnBatches/s = 7.41 avgBatchSize = 2.92 (368.8 secs) (EloDiff -110)


Based on some test data, each speed doubling gains perhaps ~250 Elo by searching deeper.
Based on some test data, each thread costs perhaps 7 Elo if using 800 visits, and 2 Elo if using 5000 visits (by making MCTS worse).
So APPROXIMATELY based on this benchmark, if you intend to do a 5 second search:
numSearchThreads =  3: (baseline)
numSearchThreads =  5:    -8 Elo
numSearchThreads =  6:   +20 Elo (recommended)
numSearchThreads =  8:    -1 Elo
numSearchThreads = 10:   -31 Elo
numSearchThreads = 12:  -110 Elo

If you care about performance, you may want to edit numSearchThreads in C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_npu_gpu_hybrid.cfg based on the above results!
If you intend to do much longer searches, configure the seconds per game move you expect with the '-time' flag and benchmark again.
If you intend to do short or fixed-visit searches, use lower numSearchThreads for better strength, high threads will weaken strength.
If interested see also other notes about performance and mem usage in the top of C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_npu_gpu_hybrid.cfg

2026-08-05 19:20:56+0800: GPU -1 finishing, processed 36059 rows 11731 batches
2026-08-05 19:20:56+0800: GPU -1 finishing, processed 10539 rows 10539 batches

PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\katago.exe benchmark -model "C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz" -config "C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_gpu.cfg"
2026-08-05 20:08:58+0800: Running with following config:
allowResignation = true
lagBuffer = 1.0
logAllGTPCommunication = true
logDir = gtp_logs
logSearchInfo = true
logSearchInfoForChosenMove = false
logToStderr = false
maxTimePondering = 60.0
maxVisits = 500
numSearchThreads = 4
onnxOpenVINODeviceType = GPU
onnxProvider = openvino
ponderingEnabled = false
resignConsecTurns = 3
resignThreshold = -0.90
rules = tromp-taylor
searchFactorAfterOnePass = 0.50
searchFactorAfterTwoPass = 0.25
searchFactorWhenWinning = 0.40
searchFactorWhenWinningThreshold = 0.95

2026-08-05 20:08:58+0800: Loading model and initializing benchmark...
2026-08-05 20:08:58+0800: Testing with default positions for board size: 19
2026-08-05 20:08:58+0800: nnRandSeed0 = 14922272731934780901
2026-08-05 20:08:58+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz useFP16 auto
2026-08-05 20:08:58+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 20:09:06+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 20:09:06+0800: ONNX backend thread 0: Model version 15
2026-08-05 20:09:06+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-fdx6d-s11272M-d5935M
2026-08-05 20:09:06+0800: ONNX backend thread 0: provider=openvino deviceIdx=GPU
2026-08-05 20:09:06+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 20:09:06+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 20:09:07+0800: ONNX backend: ONNX graph built (930658630 bytes)
2026-08-05 20:09:07+0800: ONNX backend: creating session...
2026-08-05 20:09:07+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=GPU
2026-08-05 20:09:13+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 20:09:13+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 20:09:13+0800: ONNX backend: session created, inputs=3 outputs=5

2026-08-05 20:09:16+0800: Loaded config C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_gpu.cfg
2026-08-05 20:09:16+0800: Loaded model C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz

Testing using 800 visits.
  If you have a good GPU, you might increase this using "-visits N" to get more accurate results.
  If you have a weak GPU and this is taking forever, you can decrease it instead to finish the benchmark faster.

You are currently using the ONNX Runtime version of KataGo.
Your GTP config is currently set to onnxProvider = openvino
OpenVINO device type = GPU
For Intel NPU, typically set onnxOpenVINODeviceType = NPU.
OpenVINO/NPU usually uses a single device; onnxDeviceToUseThread* is typically for cuda/trt/migraphx providers.

Your GTP config is currently set to use numSearchThreads = 4
Automatically trying different numbers of threads to home in on the best (board size 19x19):

2026-08-05 20:09:16+0800: GPU -1 finishing, processed 5 rows 5 batches
2026-08-05 20:09:16+0800: nnRandSeed0 = 4338244724291888651
2026-08-05 20:09:16+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz useFP16 auto
2026-08-05 20:09:16+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-05 20:09:34+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-05 20:09:34+0800: ONNX backend thread 0: Model version 15
2026-08-05 20:09:34+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-fdx6d-s11272M-d5935M
2026-08-05 20:09:34+0800: ONNX backend thread 0: provider=openvino deviceIdx=GPU
2026-08-05 20:09:34+0800: ONNX backend: building ONNX graph from model weights...
2026-08-05 20:09:34+0800: Building internal onnx model, requireExactNNLen=false transformerNHWC=true
2026-08-05 20:09:37+0800: ONNX backend: ONNX graph built (930658630 bytes)
2026-08-05 20:09:37+0800: ONNX backend: creating session...
2026-08-05 20:09:38+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=GPU
2026-08-05 20:09:53+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-05 20:09:53+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-05 20:09:53+0800: ONNX backend: session created, inputs=3 outputs=5


Possible numbers of threads to test: 1, 2, 3, 4, 5, 6, 8, 10, 12, 16, 20, 24, 32,

numSearchThreads =  5: 10 / 10 positions, visits/s = 22.60 nnEvals/s = 20.75 nnBatches/s = 5.57 avgBatchSize = 3.73 (355.8 secs)
numSearchThreads = 12: 10 / 10 positions, visits/s = 19.97 nnEvals/s = 19.65 nnBatches/s = 1.84 avgBatchSize = 10.67 (406.1 secs)
numSearchThreads =  3: 10 / 10 positions, visits/s = 19.09 nnEvals/s = 17.57 nnBatches/s = 10.07 avgBatchSize = 1.74 (420.2 secs)
numSearchThreads =  6: 10 / 10 positions, visits/s = 24.76 nnEvals/s = 23.35 nnBatches/s = 4.91 avgBatchSize = 4.75 (325.1 secs)
numSearchThreads =  8: 10 / 10 positions, visits/s = 24.67 nnEvals/s = 23.53 nnBatches/s = 3.45 avgBatchSize = 6.82 (327.0 secs)
numSearchThreads =  4: 10 / 10 positions, visits/s = 26.62 nnEvals/s = 24.16 nnBatches/s = 8.23 avgBatchSize = 2.94 (301.7 secs)


Ordered summary of results:

numSearchThreads =  3: 10 / 10 positions, visits/s = 19.09 nnEvals/s = 17.57 nnBatches/s = 10.07 avgBatchSize = 1.74 (420.2 secs) (EloDiff baseline)
numSearchThreads =  4: 10 / 10 positions, visits/s = 26.62 nnEvals/s = 24.16 nnBatches/s = 8.23 avgBatchSize = 2.94 (301.7 secs) (EloDiff +110)
numSearchThreads =  5: 10 / 10 positions, visits/s = 22.60 nnEvals/s = 20.75 nnBatches/s = 5.57 avgBatchSize = 3.73 (355.8 secs) (EloDiff +39)
numSearchThreads =  6: 10 / 10 positions, visits/s = 24.76 nnEvals/s = 23.35 nnBatches/s = 4.91 avgBatchSize = 4.75 (325.1 secs) (EloDiff +61)
numSearchThreads =  8: 10 / 10 positions, visits/s = 24.67 nnEvals/s = 23.53 nnBatches/s = 3.45 avgBatchSize = 6.82 (327.0 secs) (EloDiff +38)
numSearchThreads = 12: 10 / 10 positions, visits/s = 19.97 nnEvals/s = 19.65 nnBatches/s = 1.84 avgBatchSize = 10.67 (406.1 secs) (EloDiff -86)


Based on some test data, each speed doubling gains perhaps ~250 Elo by searching deeper.
Based on some test data, each thread costs perhaps 7 Elo if using 800 visits, and 2 Elo if using 5000 visits (by making MCTS worse).
So APPROXIMATELY based on this benchmark, if you intend to do a 5 second search:
numSearchThreads =  3: (baseline)
numSearchThreads =  4:  +110 Elo (recommended)
numSearchThreads =  5:   +39 Elo
numSearchThreads =  6:   +61 Elo
numSearchThreads =  8:   +38 Elo
numSearchThreads = 12:   -86 Elo

If you care about performance, you may want to edit numSearchThreads in C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_gpu.cfg based on the above results!
If you intend to do much longer searches, configure the seconds per game move you expect with the '-time' flag and benchmark again.
If you intend to do short or fixed-visit searches, use lower numSearchThreads for better strength, high threads will weaken strength.
If interested see also other notes about performance and mem usage in the top of C:\Users\foxrain\Downloads\katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64\gtp_gpu.cfg

2026-08-05 20:45:37+0800: GPU -1 finishing, processed 45325 rows 12175 batches

Looong01 · 2026-08-06T00:13:55Z

@foxrainowo Thanks again for the detailed testing — your logs (plus a day of controlled experiments on my machine) let us root-cause the NPU crash, and the fixes are now in. Updated report:

Why `gtp_npu.cfg` crashed on NPU (not a code bug)

The ZE_RESULT_ERROR_DEVICE_LOST was not a bug in KataGo's code — it was a mismatch between runtime configuration and hardware characteristics:

b11/b40-class (v17) models contain real attention layers (MatMul/Transpose/Softmax-heavy). The shared ONNX builder (from upstream PR Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) #1222) can only express those as data-dependent MatMul attention — it cannot fold them into static-weight convolutions the way older conv-only models (e.g. b28 "nbt" blocks, no attention) map. Intel's NPU compiler/driver is unstable on exactly this graph class.
Verified by isolation: the old v1.16.4 binary keeps working when we swap in the new ORT 1.28 + OpenVINO 2026.3 DLLs; the same graph runs fine on CPU and on the Intel GPU. Two failure modes on NPU: batch >= 6 -> compilation hangs forever (pure-OpenVINO repro, >60 min), and batch varying 1-5 -> intermittent DEVICE_LOST (4/4 runs, 11s to 10min). Batch pinned to 1 is always stable (your 34-min hybrid run; a 4200-inference pure-OpenVINO batch-1 loop; and a dedicated 32-min NPU-only batch-1 KataGo run, all clean).
NPU driver was already latest (32.0.100.4841). I will file this upstream to openvinotoolkit/openvino with the pure-OpenVINO repro.

Why GPU-only beat hybrid on your machine

Expected, two reasons: (1) your iGPU simply happens to be much stronger than the NPU for these models (~30x: 80 vs 2.7 visits/s in your own numbers), and (2) as above, v17 attention graphs can't be compiled into the NPU's preferred conv form, so the NPU contributes little. The NPU is a latency/efficiency-oriented accelerator (small MAC array, batch 1 already saturates it, batch N costs ~N latency for ~zero throughput gain) — that's why the hybrid config pins the NPU thread to batch 1 and lets the GPU thread do bulk batching. gtp_gpu.cfg >= gtp_npu_gpu_hybrid is the expected outcome here; hybrid/NPU only makes sense for smaller models or power-constrained scenarios.

Fixes shipped in the release configs (crash + perf regression)

gtp_npu.cfg: pinned nnMaxBatchSize = 1 (the stable regime; NPU loses nothing by it).
gtp_npu_gpu_hybrid.cfg: NPU thread stays pinned to batch 1; GPU thread unpinned (falls back to the global default).
Also: avoid benchmark autotune on NPU (it scales to 32 threads -> batch 32 -> hits the compile hang); use fixed -threads.

New backend work: making v17 NPU-friendly (+78% NPU throughput)

I extended the ONNX backend/builder with an NPU-conditional build path: when the session targets an NPU device, it can now emit a mask-free exact-board graph — the InputMask input path and all 33 attention mask-bias Add nodes ([1,heads,361,361] each, ~6-20% of NPU runtime, mathematically no-ops when the board exactly matches the network size) are dropped. Non-NPU targets build exactly as before.

Measured on b11 + NPU, batch 1: 2.73 -> 4.87 visits/s (+78%), 14 minutes / 3480 inferences with zero crashes, and fixed-seed numerical comparison vs the masked graph: 0.03c winrate / 0.4% policy delta (FP16 noise), same move choice and PV.

⚠️ Important caveat — read before using: without the mask, the net can no longer distinguish a padded board from a real one. For workloads where queries are smaller than the network size (e.g. analyzing 13x13 games on a 19x19 model), results would be silently wrong — not slower, wrong. So this is strictly opt-in: new config key onnxOpenVINONPUExactBoard (default false). It is enabled only in gtp_npu.cfg, where GTP play always uses the exact board size (the evaluator is recreated on any board-size change). Do NOT enable it for mixed-size analysis workloads.

Also fixed: OpenVINO compile cache never being created

You noticed the cache wasn't generated under gtp. Root cause: the backend left cache_dir unset by default, so every startup paid a full NPU recompile. The backend now caches compiled blobs to KataGo's standard data dir (<katago.exe dir>/KataGoData/openvino_cache on Windows, ~/.katago/openvino_cache on Linux; onnxOpenVINOCacheDir still overrides). First run writes the blob (~140 MB), later startups load it immediately.

Additionally, a new onnxOpenVINOLoadConfig key passes arbitrary OpenVINO device config (JSON) through to ONNX Runtime; the shipped configs use it to enable NPU compiler optimization-level=2 + performance-hint-override=latency (measured faster; NPU_TURBO deliberately left out — it was slower in testing).

It is here: https://github.com/Looong01/KataGo-Multi-backends/releases/tag/v1.17.1-openvino

Looong01 · 2026-08-06T00:16:28Z

@lightvector Could u pls check this? Update to v1.17.1 and NPU, GPU, Hybird mode fully tested. It's ready to merge.

foxrainowo · 2026-08-09T17:53:36Z

After updating the versions of onnxruntime and openvino to the latest versions, the NPU backend can run normally (without getting stuck), but the speed is extremely slow. In version 1.17.1, the performance of the GPU backend is excellent, with a speed of around 80v/s. I don't know what special optimizations are involved, but the result is really surprising. However, the performance of the NPU backend is very poor.

For comparison, b28_NPU_1.16.4 runs at 70v/s, while b40_NPU_1.16.4 runs at 30v/s. Assuming b11 >≈ b40, the speed of b11_NPU_1.17.1 should be greater than 30v/s.

Version	b11_NPU	b11_GPU	b40_NPU	b40_GPU
katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64-2	12 visits/s	80 visits/s (threads=5)	15 visits/s	20 visits/s
katago-v1.17.1-onnxruntime1.28.0-openvino2026.3-windows-x64	9 visits/s	80 visits/s (threads=10/12)	11 visits/s	25 visits/s
katago-v1.16.4-openvino2026.0-npu-windows-x64	Not supported	Not supported	32 visits/s	Not supported

Looong01 and others added 26 commits July 28, 2025 20:13

Add ROCm backend

1f2ae46

Fix bugs

b455530

Update

8b30cb9

Fix bugs

570ced0

Fix bugs

abb6124

All bug fixed

bfb292e

Update

4606424

test new method

1e8ea78

Update

c1a09cf

Test finished

0957b88

Update docks

c70d841

Update gitignore

1d05ca8

Update new method

9d4662b

Optimize performance

d40bd50

Update new Convlayer method

158d24d

Merge branch 'master' of https://github.com/Looong01/KataGo-ROCm

ec32eb1

Add new compile target

0bfe0a1

Merge branch 'lightvector:master' into master

f5fbb33

Add ROCm for Windows support

26d8c5b

Merge branch 'lightvector:master' into master

555d2f1

Merge branch 'lightvector:master' into master

dbc7cfa

Fix bugs

ed396b7

Merge branch 'lightvector:master' into master

ccec62c

Add Intel NPU support

ce2c9fc

Resume gitignore

d828f21

Edit README.md and Compiling.md

358dd84

Looong01 added 2 commits March 17, 2026 07:29

Add Linux Intel NPU support

496bb96

Edit Compiling.md

115e6da

Looong01 and others added 2 commits April 19, 2026 06:31

Merge branch 'lightvector:master' into Intel_NPU

f467227

Remove AMD GPU support

ffc3fef

Update openvino to 2026.1 and onnxruntime to 1.24.4

6a6a8b1

Looong01 and others added 4 commits May 12, 2026 01:14

Fix a bug

adcbf04

Merge branch 'master' into Intel_NPU

bd851d0

Update ONNX backend for Intel NPU

70dcf8d

Update ONNX backend

b37aa25

kaorahi mentioned this pull request Aug 1, 2026

Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) #1222

Open

wimi321 mentioned this pull request Aug 2, 2026

[路线] 跟进 KataGo ONNX 通用后端并重新评估 OpenCL 投入 wimi321/lizzieyzy-next#179

Open

Merge branch 'master' into Intel_NPU

fbe57ae

Looong01 changed the title ~~[Intel NPU] Add Windows & Linux Intel NPU support~~ [Intel NPU/GPU] Add Windows & Linux Intel NPU & GPU support Aug 5, 2026

Update to v1.17.1

cb629f4

Fix a little bug

a4ac35b

Conversation

Looong01 commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What Changed

1) ONNX backend and OpenVINO provider support

2) exportonnx command behavior

3) Config safety for non-ONNX binaries

4) CMake dependency flow

5) Documentation updates

Behavior Notes

Validation

Uh oh!

Looong01 commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Looong01 commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Looong01 commented Apr 19, 2026

Uh oh!

lightvector commented Apr 20, 2026

Uh oh!

Looong01 commented Apr 20, 2026

Uh oh!

kaorahi commented May 29, 2026

Uh oh!

foxrainowo commented Jun 17, 2026

Uh oh!

Looong01 commented Jun 17, 2026

Uh oh!

foxrainowo commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seniorfish commented Jul 28, 2026

Uh oh!

foxrainowo commented Aug 1, 2026

Uh oh!

Looong01 commented Aug 5, 2026

Summary

Changes

Testing

Uh oh!

Looong01 commented Aug 5, 2026

Uh oh!

foxrainowo commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Looong01 commented Aug 6, 2026

Why gtp_npu.cfg crashed on NPU (not a code bug)

Why GPU-only beat hybrid on your machine

Fixes shipped in the release configs (crash + perf regression)

New backend work: making v17 NPU-friendly (+78% NPU throughput)

Also fixed: OpenVINO compile cache never being created

Uh oh!

Looong01 commented Aug 6, 2026

Uh oh!

foxrainowo commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Looong01 commented Mar 16, 2026 •

edited

Loading

2) `exportonnx` command behavior

Looong01 commented Mar 16, 2026 •

edited

Loading

Looong01 commented Mar 16, 2026 •

edited

Loading

foxrainowo commented Jul 20, 2026 •

edited

Loading

foxrainowo commented Aug 5, 2026 •

edited

Loading

Why `gtp_npu.cfg` crashed on NPU (not a code bug)

foxrainowo commented Aug 9, 2026 •

edited

Loading