[Vulkan] dGPU + iGPU is slower in LLMs than 8GB dGPU with CPU overflow?

[NintendoManiac64]

Describe the Issue
Following the advice in #2243 I went and tried the dGPU + iGPU combination in hopes it'd perform better than the default mode of dGPU with CPU overflow since iGPU-only is definitely faster than CPU-only.

However, despite iGPU by itself being substantually faster than CPU by itself, the default dGPU by itself but with CPU overflow was still decently faster than dGPU + iGPU? Changing the value for "Main GPU" made no performance difference (I tied a value of 0, 1, and 2) nor did changing the "SplitMode" setting (though setting it to "tensor" straight up crashed). I also used nvtop to confirm that both the iGPU and dGPU were being used (results were considerably faster than the iGPU by itself anyway).

All stock settings were used other than setting CPU threads to 8 (because 8 was around 10% faster than the default 7 threads in CPU-only benchmarking) and other than setting the Vulkan devices to "all" and then manually specifying Vulkan0,Vulkan1

Additional Information:

Hardware is a Ryzen 5800H with Radeon RX 6600M 8GB (note that it's a mini PC so, much like a desktop PC, the discrete RX 6600M is used as the primary GPU).

OS was live ISO of openSUSE Tumbleweed Xfce build 2026-05-31 with GRUB boot parameter ttm.pages_limit=3840000 (I also ran sudo zypper install libvulkan_radeon vulkan-tools once booted in order to avoid issue #2102).

LLM model used (10GB); all 41 layers can fit into system (iGPU) RAM but only 26 layers fit into the 8GB dGPU VRAM: https://huggingface.co/XeyonAI/Mistral-Helcyon-Saturn-RP-12b-v1.0-GGUF/blob/main/helcyon-saturn-RP-v1.0-Q6_K.gguf

Performance Results via "Run Benchmark"

_______ RX 6600M 8GB + 5800H iGPU (3,1 tensor split; manually-specified 41 GPU layers) _______
ProcessingTime: 52.055s
ProcessingSpeed: 155.45T/s
GenerationTime: 10.717s
GenerationSpeed: 9.33T/s
TotalTime: 62.772s

_______ RX 6600M 8GB with Ryzen CPU overflow _______
ProcessingTime: 70.310s
ProcessingSpeed: 115.09T/s
GenerationTime: 15.211s
GenerationSpeed: 6.57T/s
TotalTime: 85.521s

_______ RX 6600M 8GB + 5800H iGPU _______
ProcessingTime: 126.467s
ProcessingSpeed: 63.99T/s
GenerationTime: 18.323s
GenerationSpeed: 5.46T/s
TotalTime: 144.790s

_______ 5800H iGPU only _______
ProcessingTime: 178.198s
ProcessingSpeed: 45.41T/s
GenerationTime: 36.066s
GenerationSpeed: 2.77T/s
TotalTime: 214.264s

_______ Ryzen CPU only (8 threads) _______
ProcessingTime: 555.117s
ProcessingSpeed: 14.58T/s
GenerationTime: 31.990s
GenerationSpeed: 3.13T/s
TotalTime: 587.107s

_______ Ryzen CPU only (7 threads) _______
ProcessingTime: 620.016s
ProcessingSpeed: 13.05T/s
GenerationTime: 31.193s
GenerationSpeed: 3.21T/s
TotalTime: 651.209s

[wbruna]

I don't think that's a bug, exactly: I believe it's a result of an interplay between the speeds of your three devices (dGPU, relatively fast CPU, relatively slow iGPU).

As I understand it (caveat: this is a very rough simplification of a very rough understanding on my part), a normal GPU inference does:
move data to GPU, process, move data to CPU

If you offload some layers, you get:
move data to GPU, process, move data to CPU, process on CPU

Note the number of data moves between devices stayed the same. But if you add a second GPU for some layers:
move data to GPU1, process, move data to GPU2, process, move data to CPU

Note you get an extra move between devices. Depending on the cost of that move (and it can be relatively costly for Vulkan), it may negate part of the benefit of using that second GPU.

Another factor is the number of layers on each device. The backend can't know a priori that the GPUs have vastly different speeds, so I'd expect the splitting would by default be either 50-50, or proportional to the available memory. But that's not what usually happens with CPU offload: you typically keep on the GPU as many layers as you can.

A quick test on my devices (RX 7600 XT paired with a 3400G iGPU - a bit weird because the iGPU is device 0 on my setup), with another model:

plain dGPU for a baseline:

• --usevulkan:
  ProcessingTime: 11.233s
  ProcessingSpeed: 720.38T/s
  GenerationTime: 2.876s
  GenerationSpeed: 34.77T/s
  TotalTime: 14.109s

• --usevulkan 0 1 --device Vulkan0,Vulkan1:
  ProcessingTime: 97.004s
  ProcessingSpeed: 83.42T/s
  GenerationTime: 15.022s
  GenerationSpeed: 6.66T/s
  TotalTime: 112.026s

In my case, I need to switch the indexes, so my dGPU will be the first - it's likely equivalent to setting the main device:

• --usevulkan 1 0 --device Vulkan0,Vulkan1:
  ProcessingTime: 88.599s
  ProcessingSpeed: 91.33T/s
  GenerationTime: 15.894s
  GenerationSpeed: 6.29T/s
  TotalTime: 104.493s

This is what happens if I balance them evenly (the iGPU advertises ~24G VRAM, versus ~16G on the dGPU):

• --usevulkan 1 0 --device Vulkan0,Vulkan1 --tensor-split 1 1
  ProcessingTime: 74.195s
  ProcessingSpeed: 109.06T/s
  GenerationTime: 14.792s
  GenerationSpeed: 6.76T/s
  TotalTime: 88.987s

Now, if I keep 75% of the model on the dGPU:

• --usevulkan 1 0 --device Vulkan0,Vulkan1 --tensor-split 3 1:
  ProcessingTime: 36.569s
  ProcessingSpeed: 221.28T/s
  GenerationTime: 9.321s
  GenerationSpeed: 10.73T/s
  TotalTime: 45.890s

And for ~83%:

• --usevulkan 1 0 --device Vulkan0,Vulkan1 --tensor-split 5 1:
  ProcessingTime: 23.146s
  ProcessingSpeed: 349.61T/s
  GenerationTime: 9.924s
  GenerationSpeed: 10.08T/s
  TotalTime: 33.070s

Just for completeness' sake, offloading a single layer to the CPU I get:

• --usevulkan 1 0 --gpulayers 32
  ProcessingTime: 11.863s
  ProcessingSpeed: 682.12T/s
  GenerationTime: 5.759s
  GenerationSpeed: 17.36T/s
  TotalTime: 17.622s

And for half the layers:

• --usevulkan 1 0 --gpulayers 17
  ProcessingTime: 23.336s
  ProcessingSpeed: 346.76T/s
  GenerationTime: 41.788s
  GenerationSpeed: 2.39T/s
  TotalTime: 65.124s

[NintendoManiac64]

Note the number of data moves between devices stayed the same. But if you add a second GPU for some layers:
move data to GPU1, process, move data to GPU2, process, move data to CPU

Oh sorry, I just realized I didn't spell it out that the entire model (10GB) is able to fit into system memory (16GB), so the CPU isn't really used at all when doing dGPU+iGPU (dGPU with CPU overflow uses each of the 8 threads at 100% utilization by comparison).

For more specifics, the example model has 41 layers, and doing dGPU with CPU overflow results in 26 layers running on the dGPU. By comparison, dGPU+iGPU has all 41 layers running on GPUs.

I've gone back and edited in mention of the layers and what fits.

The backend can't know a priori that the GPUs have vastly different speeds,

But it does know which devices are iGPUs because the "All" setting by default ignores iGPUs...but I suppose setting Vulkan0,Vulkan1 to override that does so by putting them on an equal playing field, thereby nullifying the dGPU/iGPU distinction to koboldcpp.

I'd expect the splitting would by default be either 50-50

I tested some more, paying specific attention to the memory allocation between the dGPU and iGPU in nvtop and, sure enough, this seemed to be what's happening! That would very much explain why changing which is the main GPU didn't make a difference because what difference does it make if its the first 50% or the last 50% on a given GPU? It's still 50% :P

That being said, my further testing led me to find what I think is still a bug with the GUI or something, but it's a different bug (may even warrant opening a new issue) that just happens to prevent me accessing the solution to my performance concerns. I see you used the command line, so this is probably why you didn't discover it, but the "Tensor Split" option in the GUI seems to only split between GPU and CPU. You'd expect putting a value of 1,1 for example would give the exact same results as leaving it blank with it running 50% on the dGPU and 50% on the iGPU, but no - it actually seems to run 50% on all GPUs (25% on dGPU, 25% on iGPU) and 50% on the CPU!

[wbruna]

Note the number of data moves between devices stayed the same. But if you add a second GPU for some layers:
move data to GPU1, process, move data to GPU2, process, move data to CPU

Oh sorry, I just realized I didn't spell it out that the entire model (10GB) is able to fit into system memory (16GB), so the CPU isn't really used at all when doing dGPU+iGPU (dGPU with CPU overflow uses each of the 8 threads at 100% utilization by comparison).

I was also considering the logits going from GPU to CPU after the last layer, before sampling.

The backend can't know a priori that the GPUs have vastly different speeds,

But it does know which devices are iGPUs because the "All" setting by default ignores iGPUs...but I suppose setting Vulkan0,Vulkan1 to override that does so by putting them on an equal playing field, thereby nullifying the dGPU/iGPU distinction to koboldcpp.

Yeah, that heuristic normally do not have to take iGPUs into account. And if you already have to fiddle with config parameters to enable them, I don't know if there is much value in trying to find a good default for that case (although I do consider "use all dGPUs first, then iGPUs" would be an improvement).

That being said, my further testing led me to find what I think is still a bug with the GUI or something, but it's a different bug (may even warrant opening a new issue) that just happens to prevent me accessing the solution to my performance concerns. I see you used the command line, so this is probably why you didn't discover it, but the "Tensor Split" option in the GUI seems to only split between GPU and CPU. You'd expect putting a value of 1,1 for example would give the exact same results as leaving it blank with it running 50% on the dGPU and 50% on the iGPU, but no - it actually seems to run 50% on all GPUs (25% on dGPU, 25% on iGPU) and 50% on the CPU!

Hmm... It should work the same; I don't see any difference in the code between setting it through the command line or through the launcher. Could you share the launching log? (I don't know if @LostRuins would prefer a separate issue)

[LostRuins]

You can use this issue, i dont really mind

[NintendoManiac64]

Could you share the launching log?

Do you just mean the terminal output? If so, see below:

no_tensor_split.txt
with_1,1_tensor_split.txt

Same setup as before except I left koboldcpp's settings at the defaults except for obviously setting GPUs to All and specifying Vulkan0,Vulkan1 (and also a second test with additionally only a tensor split of 1,1 which, again, would logically give the same results but yet...it doesn't)

Spoiler alert - the following appears in with_1.1_tensor_split.txt beginning at line 106:

WARNING: Requested buffer size (5024153632) exceeds device max_buffer_size limit (4294967292)!
load_tensors: offloading output layer to GPU
load_tensors: offloading 20 repeating layers to GPU
load_tensors: offloaded 21/41 layers to GPU
load_tensors:      Vulkan0 model buffer size =  2346.52 MiB
load_tensors:      Vulkan1 model buffer size =  2444.90 MiB
load_tensors:  Vulkan_Host model buffer size =  4791.41 MiB

And another spoiler in with_1.1_tensor_split.txt is this beginning at line 132:

llama_kv_cache:        CPU KV buffer size =   660.00 MiB
llama_kv_cache:    Vulkan0 KV buffer size =   363.00 MiB
llama_kv_cache:    Vulkan1 KV buffer size =   297.00 MiB

[wbruna]

Looks like the default splitting is proportional to each device's VRAM (roughly 1:2 in your case):

--- no_tensor_split.txt 2026-06-09 08:22:14.976741669 -0300
+++ with_1.1_tensor_split.txt   2026-06-09 08:22:21.860306054 -0300
@@ -105,13 +102,15 @@
 print_info: EOG token             = 2 '</s>'
 print_info: max token length      = 150
 load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
+
+WARNING: Requested buffer size (5024153632) exceeds device max_buffer_size limit (4294967292)!
 load_tensors: offloading output layer to GPU
-load_tensors: offloading 39 repeating layers to GPU
-load_tensors: offloaded 41/41 layers to GPU
-load_tensors:      Vulkan0 model buffer size =  3199.80 MiB
-load_tensors:      Vulkan1 model buffer size =  5858.03 MiB
-load_tensors:  Vulkan_Host model buffer size =   525.00 MiB
-...........................................................................................
+load_tensors: offloading 20 repeating layers to GPU
+load_tensors: offloaded 21/41 layers to GPU
+load_tensors:      Vulkan0 model buffer size =  2346.52 MiB
+load_tensors:      Vulkan1 model buffer size =  2444.90 MiB
+load_tensors:  Vulkan_Host model buffer size =  4791.41 MiB
+............................................................................................
 
 This model does not use SWA
 Automatic RoPE Scaling: Using model internal value.

llama_prepare_model_devices: using device Vulkan0 (AMD Radeon RX 6600M (RADV NAVI23)) (0000:03:00.0) - 8020 MiB free
llama_prepare_model_devices: using device Vulkan1 (AMD Radeon Graphics (RADV RENOIR)) (0000:06:00.0) - 15464 MiB free

And yeah, looks like --tensor-split is interfering with the autofit code, too:

@@ -39,13 +38,11 @@
 
 Overriding with 2 devices...
 
-Attempting to use llama.cpp's automating fitting code. This will override all your layer configs, may or may not work!
-Autofit Reserve Space: 1024 MB
-Autofit Success: 1, Autofit Result: -c 8320 -ngl -1
+Applying Tensor Split...
 llama_model_loader: loaded meta data with 31 key-value pairs and 363 tensors from /home/linux/helcyon-saturn-RP-v1.0-Q6_K.gguf (version GGUF V3 (latest))
 print_info: file format = GGUF V3 (latest)
 print_info: file size   = 9.36 GiB (6.56 BPW) 
-llama_prepare_model_devices: using device Vulkan0 (AMD Radeon RX 6600M (RADV NAVI23)) (0000:03:00.0) - 8020 MiB free
+llama_prepare_model_devices: using device Vulkan0 (AMD Radeon RX 6600M (RADV NAVI23)) (0000:03:00.0) - 8018 MiB free
 llama_prepare_model_devices: using device Vulkan1 (AMD Radeon Graphics (RADV RENOIR)) (0000:06:00.0) - 15464 MiB free
 init_tokenizer: initializing tokenizer for type 2
 load: 0 unused tokens

What happens with --tensor-split 2 1 or --tensor-split 5 2?

[NintendoManiac64]

Oh, whoops I did tensor split of 3,2 and 5,2, but I suppose that's not an issue since, spoiler alert - the CPU is still getting half of the layers. :P

3,2.txt
5,2.txt

I'm guessing you aren't seeing this because your using 16GB system RAM and a 16GB dGPU? I must imagine your 3400G is using dual-channel RAM so, outside of the risk of simply running out of RAM, couldn't you just remove a single RAM stick to simulate the situation I'm running into? Heck, you yourself even said your iGPU is the primary, so that'd be the same as me with 8GB primary + 16GB secondary (just with the iGPU/dGPU positions reversed).

[wbruna]

I'm guessing you aren't seeing this because your using 16GB system RAM and a 16GB dGPU?

48G system RAM, actually, with 2G allocated for video (so around 25G available on the iGPU).

What I didn't do was enabling autofit, simply because there was no point in testing a different model in a system with different specs 🙂 I was focused on comparing layers-on-CPU versus layers-on-iGPU.

So, I see more than one question here:

• if you configure your instance to manually place the same number of offloaded layers on the iGPU instead of on the CPU, does it get faster?
• could the autofit code apply better heuristics when dealing with more than one GPU / explicit tensor splitting?

[NintendoManiac64]

48G system RAM

I'm guessing the idea is that you're using a 3400G so that you can use the iGPU for desktop use while the dGPU can be purely dedicated to koboldcpp workloads?

　

What I didn't do was enabling autofit
if you configure your instance to manually place the same number of offloaded layers on the iGPU instead of on the CPU, does it get faster?

I don't know if this is what you were suggesting, but I got things to work! The GUI's GPU Layers option needs to be changed from its default of -1 to however many layers the model your using has (41 in my case); sorted from fastest to slowest:

3.1_tensor-split+2threads@41Layers.txt
3,1_tensor-split@41Layers.txt
5,2_tensor-split@41Layers.txt
7,2_tensor-split@41Layers.txt
2,1_tensor-split@41Layers.txt
5,3_tensor-split@41Layers.txt
3,2_tensor-split@41Layers.txt
1,1_tensor-split.txt
1,1_tensor-split@41Layers.txt
2,3_tensor-split@41Layers.txt
4,1_tensor-split@41Layers=crash.txt

_______ RX 6600M 8GB + 5800H iGPU (3,1 tensor split; manually-specified 41 GPU layers) _______
ProcessingTime: 52.055s
ProcessingSpeed: 155.45T/s
GenerationTime: 10.717s
GenerationSpeed: 9.33T/s
TotalTime: 62.772s

(you can tell it's properly working because even with only 2 CPU threads the performance is within margin of error relative to 7 CPU threads - heck the 2 CPU threads was even a teeny bit faster and is the result above :P Though I didn't see if this was a repeatable difference)

　
And while that is a solution, that really doesn't seem like the solution as it requires manually specifying the total layer amount which can obviously differ from model to model. Again, seems like there's an issue with autofit.

　

UPDATE

＊sigh＊ If only it were that simple...

I tried a bit larger model (13GB) from the same group:

• https://huggingface.co/XeyonAI/Mistral-Helcyon-Nebula-12b-v2.0-GGUF/blob/main/helcyon-nebula-v2.0-Q8_0.gguf

And while performance behavior initially seemed the same, I don't gain enough performance back from fully using dGPU+iGPU rather than dGPU with CPU overflow...

Worse still, the model seems to be too big and I don't know of Vulkan-supporting Linux OSes with desktop RAM usage slim enough to create baseline CPU-only or iGPU-only benchmarks, and it's not exactly the best of times to be buying more RAM either...

Regardless, here's my terminal output from that 13GB model sorted from fastest to slowest:

dGPU_only.txt
3,2@41Layers.txt
4,3@41Layers.txt
2,1.txt
3,2.txt
1,1.txt
1,1@41Layers.txt
2,3.txt
Vulkan0,Vulkan1 (blank tensor split).txt
7,4@41Layers=crash.txt