diff --git a/.gitignore b/.gitignore index 7799d05291..64957937ea 100644 --- a/.gitignore +++ b/.gitignore @@ -32,7 +32,7 @@ cpp/tests/results/matchsgfs2/games.sgfs cpp/data/ versions/ -cpp/build +cpp/build* cpp/out export_model_cuda.sh @@ -65,7 +65,7 @@ CMakeCache.txt CMakeFiles/ Makefile cmake_install.cmake -cpp/builds/ +cpp/builds*/ .DS_Store GPATH @@ -74,6 +74,7 @@ GTAGS cpp/external/httplib/cpp-httplib/ cpp/external/nlohmann_json/nlohmann_json +cpp/external/onnxruntime-win-x64-openvino gtp.cfg katago_contribute/ tmpsgf/ diff --git a/Compiling.md b/Compiling.md index abe7de36fc..26dcc8efbd 100644 --- a/Compiling.md +++ b/Compiling.md @@ -34,6 +34,7 @@ As also mentioned in the instructions below but repeated here for visibility, if * If using the CUDA backend, CUDA 11 or later and a compatible version of CUDNN based on your CUDA version (https://developer.nvidia.com/cuda-toolkit) (https://developer.nvidia.com/cudnn) and a GPU capable of supporting them. * If using the TensorRT backend, in addition to a compatible CUDA Toolkit (https://developer.nvidia.com/cuda-toolkit), you also need TensorRT (https://developer.nvidia.com/tensorrt) that is at least version 8.5. * If using the Eigen backend, Eigen3. With Debian packages, (i.e. apt or apt-get), this should be `libeigen3-dev`. + * If using the ONNX backend, ONNX Runtime headers/libs, plus Protobuf (for generating the vendored ONNX schema used for `.bin.gz` model conversion -- the same dependency the TensorRT backend uses). * zlib, libzip. With Debian packages (i.e. apt or apt-get), these should be `zlib1g-dev`, `libzip-dev`. * If you want to do self-play training and research, probably Google perftools `libgoogle-perftools-dev` for TCMalloc or some other better malloc implementation. For unknown reasons, the allocation pattern in self-play with large numbers of threads and parallel games causes a lot of memory fragmentation under glibc malloc that will eventually run your machine out of memory, but better mallocs handle it fine. * If compiling to contribute to public distributed training runs, OpenSSL is required (`libssl-dev`). @@ -41,7 +42,7 @@ As also mentioned in the instructions below but repeated here for visibility, if * `git clone https://github.com/lightvector/KataGo.git` * Compile using CMake and make in the cpp directory: * `cd KataGo/cpp` - * `cmake . -DUSE_BACKEND=OPENCL` or `cmake . -DUSE_BACKEND=CUDA` or `cmake . -DUSE_BACKEND=TENSORRT` or `cmake . -DUSE_BACKEND=EIGEN` depending on which backend you want. + * `cmake . -DUSE_BACKEND=OPENCL` or `cmake . -DUSE_BACKEND=CUDA` or `cmake . -DUSE_BACKEND=TENSORRT` or `cmake . -DUSE_BACKEND=EIGEN` or `cmake . -DUSE_BACKEND=ONNX` depending on which backend you want. * Specify also `-DUSE_TCMALLOC=1` if using TCMalloc. * Compiling will also call git commands to embed the git hash into the compiled executable, specify also `-DNO_GIT_REVISION=1` to disable it if this is causing issues for you. * Specify `-DUSE_AVX2=1` to also compile Eigen with AVX2 and FMA support, which will make it incompatible with old CPUs but much faster. (If you want to go further, you can also add `-DCMAKE_CXX_FLAGS='-march=native'` which will specialize to precisely your machine's CPU, but the exe might not run on other machines at all). @@ -54,6 +55,49 @@ As also mentioned in the instructions below but repeated here for visibility, if * You will probably want to edit `configs/gtp_example.cfg` (see "Tuning for Performance" above). * If using OpenCL, you will want to verify that KataGo is picking up the correct device when you run it (e.g. some systems may have both an Intel CPU OpenCL and GPU OpenCL, if KataGo appears to pick the wrong one, you can correct this by specifying `openclGpuToUse` in `configs/gtp_example.cfg`). +##### ONNX Runtime Backend (Linux) +The ONNX backend uses ONNX Runtime for inference, and supports both: +* `.onnx` models loaded directly. +* `.bin.gz` KataGo models via internal conversion to ONNX graph (requires ONNX protobuf dependencies in CMake). + +##### Linux Intel NPU (OpenVINO EP) Setup +1. Install Intel NPU driver on Linux: + * https://github.com/intel/linux-npu-driver +2. Install OpenVINO via system package manager (APT example): + * https://docs.openvino.ai/2025/get-started/install-openvino/install-openvino-apt.html +3. Build ONNX Runtime with OpenVINO EP for NPU (same ORT flow as Windows): + * https://onnxruntime.ai/docs/build/eps.html#openvino + * Set OpenVINO EP build option so `use_openvino` is `NPU` (for example `--use_openvino NPU` in ORT build.py). + * For instance, `./build.sh --config Release --use_openvino NPU --build_shared_lib --skip_tests --parallel --cmake_extra_defines CMAKE_INSTALL_PREFIX=/cpp/external/onnxruntime-win-x64-openvino`. + +##### Prepare `ONNXRUNTIME_ROOT` in KataGo (Linux) +Install package root: +* `cpp/external/onnxruntime-linux-x64-openvino` by running `cmake --install build\Linux\Release --config Release`. + +##### Minimal KataGo Build Commands (Linux, ONNX backend) +On Linux, `KATAGO_AUTO_FETCH_DEPS=ON` can auto-fetch missing `zlib` and `protobuf` dependencies via vcpkg into `cpp/build/deps/vcpkg`. The ONNX graph itself is generated from the vendored `cpp/external/onnx/onnx.proto` schema at build time (via `protobuf_generate_cpp`), the same mechanism the TensorRT backend uses for its ONNX-emitter path, so no separate `onnx` package is needed. + +```bash +cmake -S cpp -B cpp/build -G Ninja -DUSE_BACKEND=ONNX -DONNXRUNTIME_ROOT=cpp/external/onnxruntime-linux-x64-openvino +cmake --build cpp/build -j +``` + +If you want to disable auto-fetch and provide dependencies manually: +* `-DKATAGO_AUTO_FETCH_DEPS=OFF` +* plus `-DProtobuf_PROTOC_EXECUTABLE=... -DProtobuf_INCLUDE_DIR=... -DProtobuf_LIBRARY=... -DZLIB_INCLUDE_DIR=... -DZLIB_LIBRARY=...` + +Typical run config for Intel NPU: +* `onnxProvider = openvino` +* `onnxOpenVINODeviceType = NPU` +* `onnxOpenVINOEnableNPUFastCompile = true` (optional; may be ignored on ORT builds that do not support this key) + +Multi-device assignment is mainly for `onnxProvider=cuda/tensorrt` (`onnxDeviceToUseThread*`). +For `onnxProvider=openvino` on Intel NPU, a single device is typically used. + +See the [README's ONNX/OpenVINO Intel NPU Quick Start](README.md#onnxopenvino-intel-npu-quick-start-linux) +for how to run on NPU only, iGPU only, or both together. + + ## Windows * TLDR: * Building from source on Windows is actually a bit tricky, depending on what version you're building, there's not necessarily a super-fast way. @@ -64,13 +108,8 @@ As also mentioned in the instructions below but repeated here for visibility, if * If using the CUDA backend, CUDA 11 or later and a compatible version of CUDNN based on your CUDA version (https://developer.nvidia.com/cuda-toolkit) (https://developer.nvidia.com/cudnn) and a GPU capable of supporting them. I'm unsure how version compatibility works with CUDA, there's a good chance that later versions than these work just as well, but they have not been tested. * If using the TensorRT backend, in addition to a compatible CUDA Toolkit (https://developer.nvidia.com/cuda-toolkit), you also need TensorRT (https://developer.nvidia.com/tensorrt) that is at least version 8.5. * If using the Eigen backend, Eigen3, version 3.3.x. (http://eigen.tuxfamily.org/index.php?title=Main_Page#Download). - * zlib. Easy way to build zlib on Windows is to use vcpkg. Run in Powershell: - * git clone https://github.com/microsoft/vcpkg.git - * cd .\vcpkg\ - * .\bootstrap-vcpkg.bat - * .\vcpkg.exe install zlib:x64-windows - * Set CMake ZLIB_LIBRARY to vcpkg\installed\x64-windows\lib\zlib.lib and ZLIB_INCLUDE_DIRECTORY to vcpkg\installed\x64-windows\include. - * Copy zlib1.dll from vcpkg\installed\x64-windows\bin to Katago folder after you've built Katago executable. + * If using the ONNX backend, ONNX Runtime package (headers + import libs + runtime DLLs). + * On Windows, missing `zlib` and `protobuf` (used to generate the vendored ONNX schema) can be auto-fetched by CMake into `cpp/build/deps/vcpkg` (default `KATAGO_AUTO_FETCH_DEPS=ON`). * libzip (optional, needed only for self-play training) - for example https://github.com/kiyolee/libzip-win-build * For MinGW it's recommended to use [MSYS2](https://www.msys2.org/) building platform to get necessary zlib and libzip dependencies: * Install MSYS2 according to the instruction on the official site @@ -97,7 +136,7 @@ As also mentioned in the instructions below but repeated here for visibility, if -DLIBZIP_INCLUDE_DIR_ZIPCONF:PATH="C:/msys64/mingw64/include" -DLIBZIP_LIBRARY:FILEPATH="C:/msys64/mingw64/lib/libzip.dll.a" ``` - * Also set `USE_BACKEND` to `OPENCL`, or `CUDA`, or `TENSORRT`, or `EIGEN` depending on what backend you want to use. + * Also set `USE_BACKEND` to `OPENCL`, or `CUDA`, or `TENSORRT`, or `EIGEN`, or `ONNX` depending on what backend you want to use. * Set any other options you want and re-run "Configure" again as needed after setting them. Such as: * `NO_GIT_REVISION` if you don't have Git or if cmake is not finding it. * `NO_LIBZIP` if you don't care about running self-play training and you don't have libzip. @@ -117,6 +156,55 @@ As also mentioned in the instructions below but repeated here for visibility, if * You will probably want to edit `configs/gtp_example.cfg` (see "Tuning for Performance" above). * If using OpenCL, you will want to verify that KataGo is picking up the correct device (e.g. some systems may have both an Intel CPU OpenCL and GPU OpenCL, if KataGo appears to pick the wrong one, you can correct this by specifying `openclGpuToUse` in `configs/gtp_example.cfg`). +##### ONNX Runtime Backend +The ONNX backend uses ONNX Runtime for inference, and supports both: +* `.onnx` models loaded directly. +* `.bin.gz` KataGo models via internal conversion to ONNX graph (requires ONNX protobuf dependencies in CMake). + +##### Windows Intel NPU (OpenVINO EP) Setup +1. Install Visual Studio 2026 Community or Visual Studio 2026 Build Tools: + * https://visualstudio.microsoft.com/zh-hans/downloads/ + * In installer workloads, select **Desktop development with C++**. +2. Install Intel NPU driver: + * https://www.intel.com/content/www/us/en/download/794734/intel-npu-driver-windows.html +3. Install OpenVINO 2026 archive package on Windows: + * https://docs.openvino.ai/2026/get-started/install-openvino/install-openvino-archive-windows.html + * Typical install root looks like: `C:\Program Files (x86)\Intel\openvino` +4. Add these to System PATH: + * `C:\Program Files (x86)\Intel\openvino\runtime\bin\intel64\Release` + * `C:\Program Files (x86)\Intel\openvino\runtime\3rdparty\tbb\bin` +5. Build ONNX Runtime with OpenVINO EP for NPU (follow official docs): + * https://onnxruntime.ai/docs/build/eps.html#openvino + * Set OpenVINO EP build option so `use_openvino` is `NPU` (for example `--use_openvino NPU` in ORT build.py). + * For example, `.\build.bat --config Release --use_openvino NPU --build_shared_lib --skip_tests --parallel --cmake_extra_defines CMAKE_INSTALL_PREFIX=\cpp\external\onnxruntime-win-x64-openvino` + +##### Prepare `ONNXRUNTIME_ROOT` in KataGo (Windows) +Install package root: +* `cpp/external/onnxruntime-win-x64-openvino` by running `cmake --install build\Windows\Release --config Release` + +##### Minimal KataGo Build Commands (Windows, ONNX backend) +On Windows, `KATAGO_AUTO_FETCH_DEPS=ON` by default, so missing `zlib` and `protobuf` dependencies are auto-fetched via vcpkg into `cpp/build/deps/vcpkg`. The ONNX graph itself is generated from the vendored `cpp/external/onnx/onnx.proto` schema at build time (via `protobuf_generate_cpp`), the same mechanism the TensorRT backend uses for its ONNX-emitter path, so no separate `onnx` package is needed. + +``` +cmake -S cpp -B cpp/build -G "Visual Studio 18 2026" -A x64 -DUSE_BACKEND=ONNX -DONNXRUNTIME_ROOT=cpp/external/onnxruntime-win-x64-openvino +cmake --build cpp/build --config Release -j +``` + +If you want to disable auto-fetch and provide dependencies manually: +* `-DKATAGO_AUTO_FETCH_DEPS=OFF` +* plus `-DProtobuf_PROTOC_EXECUTABLE=... -DProtobuf_INCLUDE_DIR=... -DProtobuf_LIBRARY=... -DZLIB_INCLUDE_DIR=... -DZLIB_LIBRARY=...` + +Typical run config for Intel NPU: +* `onnxProvider = openvino` +* `onnxOpenVINODeviceType = NPU` +* `onnxOpenVINOEnableNPUFastCompile = true` (optional; may be ignored on ORT builds that do not support this key) + +Multi-device assignment is mainly for `onnxProvider=cuda/tensorrt` (`onnxDeviceToUseThread*`). +For `onnxProvider=openvino` on Intel NPU, a single device is typically used. + +See the [README's ONNX/OpenVINO Intel NPU Quick Start](README.md#onnxopenvino-intel-npu-quick-start-windows) +for how to run on NPU only, iGPU only, or both together. + ## MacOS * TLDR (Metal backend - recommended for most users, hybrid CPU+GPU+Neural Engine for maximum throughput): ``` diff --git a/README.md b/README.md index 4ea665e7f5..92c2de642f 100644 --- a/README.md +++ b/README.md @@ -1,27 +1,32 @@ # KataGo -* [Overview](#overview) -* [Training History and Research](#training-history-and-research) -* [Where To Download Stuff](#where-to-download-stuff) -* [Setting Up and Running KataGo](#setting-up-and-running-katago) - * [GUIs](#guis) - * [Windows and Linux](#windows-and-linux) - * [MacOS](#macos) - * [OpenCL vs CUDA vs TensorRT vs Eigen](#opencl-vs-cuda-vs-tensorrt-vs-eigen) - * [How To Use](#how-to-use) - * [Tuning for Performance](#tuning-for-performance) - * [Common Questions and Issues](#common-questions-and-issues) - * [Issues with specific GPUs or GPU drivers](#issues-with-specific-gpus-or-gpu-drivers) - * [Common Problems](#common-problems) - * [Other Questions](#other-questions) -* [Features for Developers](#features-for-developers) - * [GTP Extensions](#gtp-extensions) - * [Analysis Engine](#analysis-engine) -* [Compiling KataGo](#compiling-katago) -* [Source Code Overview](#source-code-overview) -* [Selfplay Training](#selfplay-training) -* [Contributors](#contributors) -* [License](#license) +- [KataGo](#katago) + - [Overview](#overview) + - [Training History and Research and Docs](#training-history-and-research-and-docs) + - [Where To Download Stuff](#where-to-download-stuff) + - [Setting Up and Running KataGo](#setting-up-and-running-katago) + - [GUIs](#guis) + - [Windows and Linux](#windows-and-linux) + - [MacOS](#macos) + - [OpenCL vs CUDA vs TensorRT vs Eigen vs ONNX](#opencl-vs-cuda-vs-tensorrt-vs-eigen-vs-onnx) + - [How To Use](#how-to-use) + - [ONNX/OpenVINO Intel NPU Quick Start (Windows)](#onnxopenvino-intel-npu-quick-start-windows) + - [ONNX/OpenVINO Intel NPU Quick Start (Linux)](#onnxopenvino-intel-npu-quick-start-linux) + - [Human-style Play and Analysis](#human-style-play-and-analysis) + - [Other Commands:](#other-commands) + - [Tuning for Performance](#tuning-for-performance) + - [Common Questions and Issues](#common-questions-and-issues) + - [Issues with specific GPUs or GPU drivers](#issues-with-specific-gpus-or-gpu-drivers) + - [Common Problems](#common-problems) + - [Other Questions](#other-questions) + - [Features for Developers](#features-for-developers) + - [GTP Extensions:](#gtp-extensions) + - [Analysis Engine:](#analysis-engine) + - [Compiling KataGo](#compiling-katago) + - [Source Code Overview:](#source-code-overview) + - [Selfplay Training:](#selfplay-training) + - [Contributors](#contributors) + - [License](#license) ## Overview @@ -85,8 +90,8 @@ The community also provides KataGo packages for [Homebrew](https://brew.sh) on M Use `brew install katago`. The latest config files and networks are installed in KataGo's `share` directory. Find them via `brew list --verbose katago`. A basic way to run katago will be `katago gtp -config $(brew list --verbose katago | grep 'gtp.*\.cfg') -model $(brew list --verbose katago | grep .gz | head -1)`. You should choose the Network according to the release notes here and customize the provided example config as with every other way of installing KataGo. -### OpenCL vs CUDA vs TensorRT vs Eigen -KataGo has four backends, OpenCL (GPU), CUDA (GPU), TensorRT (GPU), and Eigen (CPU). (On macOS there is also a Metal backend, most easily obtained via homebrew - see above.) +### OpenCL vs CUDA vs TensorRT vs Eigen vs ONNX +KataGo has five backends, OpenCL (GPU), CUDA (GPU), TensorRT (GPU), Eigen (CPU), and ONNX (CPU/GPU/NPU via providers).(On macOS there is also a Metal backend, most easily obtained via homebrew - see above.) As of v1.17, KataGo supports transformer neural nets, which are generally much stronger for the same compute cost and which the main training run is switching to. Transformer models are more demanding on the GPU backend than the older convolutional nets, so the backend recommendations below matter more for them - in particular OpenCL is noticeably slower on transformers, and on NVIDIA the CUDNN and TensorRT versions make a large difference. @@ -96,12 +101,14 @@ The quick summary is: * Use Eigen with AVX2 if you don't have a GPU or if your GPU is too old/weak to work with OpenCL, and you just want a plain CPU KataGo. * Use Eigen without AVX2 if your CPU is old or on a low-end device that doesn't support AVX2. * The CUDA+CUDNN backend can also work well for NVIDIA GPUs. It has faster startup than TensorRT and is competitive on transformers if using CUDNN >= 9.8.0, though TensorRT 10.16 is often still slightly faster. + * ONNX backend uses ONNX Runtime execution providers (CPU/OpenVINO/CUDA/TensorRT/MIGraphX/CoreML). It is useful for Intel NPU (OpenVINO) and raw `.onnx` models. More in detail: * OpenCL is a general GPU backend should be able to run with any GPUs or accelerators that support [OpenCL](https://en.wikipedia.org/wiki/OpenCL), including NVIDIA GPUs, AMD GPUs, as well CPU-based OpenCL implementations or things like Intel Integrated Graphics. This is the most general GPU version of KataGo and doesn't require a complicated install like CUDA does, so is most likely to work out of the box as long as you have a fairly modern GPU. **However, it also need to take some time when run for the very first time to tune itself.** For many systems, this will take 5-30 seconds, but on a few older/slower systems, may take many minutes or longer. Also, the quality of OpenCL implementations is sometimes inconsistent, particularly for Intel Integrated Graphics and for AMD GPUs that are older than several years, so it might not work for very old machines, as well as specific buggy newer AMD GPUs, see also [Issues with specific GPUs or GPU drivers](#issues-with-specific-gpus-or-gpu-drivers). OpenCL is not as optimized as the NVIDIA-specific backends and will generally be slower, particularly for transformer models. * CUDA is a GPU backend specific to NVIDIA GPUs (it will not work with AMD or Intel or any other GPUs) and requires installing [CUDA](https://developer.nvidia.com/cuda-zone) and [CUDNN](https://developer.nvidia.com/cudnn) and a modern NVIDIA GPU. For older convolutional nets, on many GPUs the OpenCL implementation can match or beat NVIDIA's own CUDA/CUDNN, with the exception of top-end NVIDIA GPUs that support FP16 and tensor cores. For transformer nets, CUDA+CUDNN clearly outperforms OpenCL, but you should use CUDNN >= 9.8.0 if at all possible - the older CUDNN 8.9.7 is a LOT slower on transformer models. Compared to TensorRT, CUDA+CUDNN has faster startup times and is often only slightly slower (and occasionally faster) on transformers. * TensorRT is similar to CUDA, but uses NVIDIA's TensorRT framework to run the neural network with more optimized kernels. For modern NVIDIA GPUs it should work whenever CUDA does, and will usually be the fastest backend, though it has much longer startup times on every launch. As of v1.17.0, TensorRT versions older than 10 are no longer supported. For transformer models, recent versions like CUDA 13 + TensorRT 10.16 are best, while older TensorRT versions can be outperformed by CUDA+CUDNN. * Eigen is a *CPU* backend that should work widely *without* needing a GPU or fancy drivers. Use this if you don't have a good GPU or really any GPU at all. It will be quite significantly slower than OpenCL or CUDA, but on a good CPU can still often get 10 to 20 playouts per second if using the smaller (15 or 20) block neural nets. Eigen can also be compiled with AVX2 and FMA support, which can provide a big performance boost for Intel and AMD CPUs from the last few years. However, it will not run at all on older CPUs (and possibly even some recent but low-power modern CPUs) that don't support these fancy vector instructions. + * ONNX backend uses [ONNX Runtime](https://onnxruntime.ai/). It can use CPU by default, OpenVINO for Intel hardware (including NPU on supported systems), CUDA/TensorRT for NVIDIA GPUs, MIGraphX for AMD GPUs, and CoreML on macOS. Multi-device assignment via `onnxDeviceToUseThread*` is mainly for CUDA/TensorRT providers, while OpenVINO NPU setups are typically single-device. For **any** implementation, it's recommended that you also tune the number of threads used if you care about optimal performance, as it can make a factor of 2-3 difference in the speed. See "Tuning for Performance" below. However, if you mostly just want to get it working, then the default untuned settings should also be still reasonable. @@ -135,6 +142,90 @@ path/to/katago.exe gtp -model path/to/.bin.gz path/to/katago.exe gtp -model path/to/.bin.gz -config path/to/gtp_custom.cfg ``` +#### ONNX/OpenVINO Intel NPU Quick Start (Windows) + +The ONNX backend loads a standard `.bin.gz` KataGo model **directly** — no conversion step needed. It can also load a raw `.onnx` file (e.g. hand-exported or quantized elsewhere, or via `exportonnx` below); see the note on that below. + +> **Note on raw `.onnx` files and model version:** When loading a raw `.onnx` model (not a KataGo `.bin.gz`), the backend auto-detects the model version from output tensor shapes. This heuristic is usually correct for standard KataGo models, but may misdetect the version for unusual configurations, causing incorrect score or ownership outputs without any error. If results look wrong, add `onnxModelVersion = ` (e.g. `onnxModelVersion = 15`) to your GTP config to force the correct version. Standard `.bin.gz` KataGo models are unaffected — the version is always read from the file directly. + +If you want to use ONNX Runtime + OpenVINO on Intel NPU: +* Install Intel NPU driver: https://www.intel.com/content/www/us/en/download/794734/intel-npu-driver-windows.html +* Install OpenVINO archive package (Windows): https://docs.openvino.ai/2026/get-started/install-openvino/install-openvino-archive-windows.html +* Typical install root looks like: `C:\Program Files (x86)\Intel\openvino` +* Add `\runtime\bin\intel64\Release` and `\runtime\3rdparty\tbb\bin` to System PATH + +Minimal commands, using the `.bin.gz` model directly: +``` +# NPU only +./katago.exe benchmark -config cpp/configs/gtp_example.cfg -model .bin.gz -override-config onnxProvider=openvino,onnxOpenVINODeviceType=NPU +./katago.exe gtp -config cpp/configs/gtp_example.cfg -model .bin.gz -override-config onnxProvider=openvino,onnxOpenVINODeviceType=NPU + +# Intel iGPU only (same as above, but device type GPU) +./katago.exe benchmark -config cpp/configs/gtp_example.cfg -model .bin.gz -override-config onnxProvider=openvino,onnxOpenVINODeviceType=GPU +``` + +NPU + iGPU running together: run two ONNX backend server threads and pin each one to a different +device (a single OpenVINO session spanning both via `onnxOpenVINODeviceType=MULTI:NPU,GPU` does not +parallelize well in practice). Pair this with a per-thread batch size override, since the NPU wants +batch 1 while the iGPU benefits from a larger batch: +``` +./katago.exe benchmark -config cpp/configs/gtp_example.cfg -model .bin.gz -override-config "onnxProvider=openvino,numNNServerThreadsPerModel=2,onnxOpenVINODeviceTypeThread0=NPU,onnxOpenVINODeviceTypeThread1=GPU,nnMaxBatchSizeThread0=1,nnMaxBatchSizeThread1=8" +``` +Check the log for `ONNX backend thread 0: provider=openvino deviceIdx=NPU` and +`ONNX backend thread 1: provider=openvino deviceIdx=GPU` to confirm the two threads actually landed +on the devices you expect. + +See the "ONNX backend settings" section of [`cpp/configs/gtp_example.cfg`](cpp/configs/gtp_example.cfg) +for the full list of `onnx*` and `nnMaxBatchSizeThread` keys, including per-device-type EP tuning +(`onnxOpenVINODeviceConfig__