The Dual-GPU Trap: When Adding a Second Graphics Card Ruins Your AI Performance
Welcome back to the Silicon Command Center. Today, we're diving into the results of a benchmarking session that perfectly illustrates one of the most counter-intuitive lessons in local AI hardware.
Last week, we got our hands on a brand new Acer Predator Orion desktop (Intel Core Ultra 7 265KF, 32GB DDR5). Out of the box, it came with a highly capable NVIDIA RTX 5070 (12GB VRAM). But we decided to get creative. We threw an aftermarket RTX 5060 (8GB VRAM) into the second PCIe slot, giving us an "asymmetric" dual-GPU setup with a combined 20GB of VRAM.
More GPUs means more performance, right? Wrong.
Let's look at the data.
Test 1: The Single GPU Baseline
First, we ran Llama-3.1-8B entirely on the primary RTX 5070. The 8B model easily fits inside the 12GB of VRAM, so the 5070 didn't need any help.
- Model: Llama-3.1-8B (Q4)
- Hardware: RTX 5070 Only
- Speed: 108.5 tokens/sec
Blazing fast. The model is entirely resident in the GPU's memory, so the tensor cores are chewing through tokens at maximum velocity.
Test 2: The Dual-GPU Trap
Next, we purposely configured Ollama to split that exact same Llama-3.1-8B model across both GPUs (the 5070 and the 5060). We have more compute power active now, so it should be faster, right?
- Model: Llama-3.1-8B (Q4)
- Hardware: RTX 5070 + RTX 5060 (Split)
- Speed: 60.1 tokens/sec
Our performance was nearly cut in half.
Why? Because of the PCIe Bus Bottleneck. When you split a model across two discrete GPUs, they have to constantly communicate and pass tensors back and forth across your motherboard's PCIe lanes. The time it takes to move data over the motherboard is significantly slower than the time it takes for a GPU to talk to its own internal VRAM.
The Lesson: Never split a model across multiple GPUs if it can fit entirely inside your primary GPU's memory. You are introducing a massive latency penalty for zero gain.
Test 3: The Spillover Cliff
So, when should you use both GPUs? When the model is too big for one! We loaded up DeepSeek-R1-32B. This model is massive, requiring more than the 20GB of VRAM our two cards provide combined.
The inference engine filled up the 5070, filled up the 5060, and then "spilled over" the remaining layers into the system's standard DDR5 RAM.
- Model: DeepSeek-R1-32B (Q4)
- Hardware: RTX 5070 + RTX 5060 + System RAM
- Speed: 5.5 tokens/sec
We hit the cliff. As we discussed in our last post, spilling into system RAM on a PC forces the GPUs to wait on the dramatically slower system memory. 5.5 tokens/sec is painful to read, but it's the price you pay for running a 32B model on a budget rig.
Test 4: The CPU Baseline
Finally, we wanted to see how the Intel Core Ultra 7 handled things completely on its own, utilizing only the system DDR5 RAM without touching the GPUs at all.
- Model: Llama-3.1-8B (Q4)
- Hardware: CPU Only
- Speed: 11.0 tokens/sec
The Takeaway: Modern CPUs are actually incredibly capable! 11 tokens per second is highly readable. If you don't have a dedicated GPU, don't let that stop you from running local AI. A good modern CPU can absolutely run 8B class models natively.
Conclusion
Our asymmetric Acer Predator Orion test rig taught us a valuable lesson: In local AI, Memory Bandwidth is king, and Data Locality is queen.
Keep your models inside your VRAM whenever possible. Only scale out to a second GPU if you desperately need the capacity, and avoid spilling into system RAM unless you have the patience of a saint.
Check out the Hardware Lab for the full breakdown of all our test rigs!