Speculative Decoding Explained: How Dual-Model Inference Doubles Local LLM Speed
If you have spent any time running local Large Language Models, you have likely encountered the fundamental speed limit of local inference: memory bandwidth.
Whether you are running an RTX 4060, an RTX 5070, or an Apple M-series Mac, upgrading to a faster GPU core doesn't scale your text generation speed as much as you'd expect. An 8B parameter model running at 4-bit quantization on an RTX 4060 tops out around 50–60 tokens/sec.
To break past that ceiling, the conventional advice has always been: quantize the model down further (e.g., Q3 or Q2) or buy a GPU with a wider memory bus. Both choices involve painful trade-offs: either you degrade the model's reasoning capability, or you empty your wallet.
Enter Speculative Decoding.
Speculative decoding is arguably the most elegant algorithmic breakthrough in modern inference runtimes. It allows you to accelerate local token generation by 1.5x to 2.5x with zero loss in precision, zero drift in perplexity, and 100% mathematical fidelity to the original model.
Here is a deep dive into how it works, why it exploits consumer GPU architecture so effectively, and how you can run it on your own hardware today.
The Root Problem: The Autoregressive Memory Wall
To understand why speculative decoding works, you first have to understand why standard LLM generation is so wildly inefficient on modern GPUs.
Modern transformer models generate text autoregressively: one token at a time. To generate a single token, the model must take all previous tokens, pass them through dozens of transformer layers, compute attention, and output a single probability distribution.
Here is the catch: To generate that single token, the GPU must read every single parameter in the model from VRAM into the compute cores.
Let's look at the math for a standard quantized 8-billion parameter model (Q4):
- Model Weight Size: ~4.6 GB
- To emit 1 token: The GPU memory controller must transfer ~4.6 GB of weights across the memory bus.
- To emit 50 tokens/sec: The GPU must transfer: 4.6 GB × 50 tokens/sec ≈ 230 GB/sec
An RTX 4060 has a maximum theoretical memory bandwidth of ~272 GB/s. In practice, after accounting for cache misses and KV cache transfers, generating at ~55 tokens/sec completely saturates the memory bus.
Meanwhile, the GPU's thousands of CUDA cores and Tensor cores—capable of hundreds of trillions of math operations per second (TFLOPS)—are sitting idle for over 90% of each generation cycle, waiting for the memory bus to feed them numbers.
Autoregressive inference is memory-bound, not compute-bound.
The Solution: The "Draft and Verify" Architecture
Speculative decoding flips this bottleneck on its head by pairing two different models together:
- The Draft Model (The Sprinter): A tiny, lightweight model (e.g., a 1B parameter model) running on the same tokenizer.
- The Target Model (The Professor): Your primary, high-quality model (e.g., an 8B or 14B model) whose intelligence and output quality you want to preserve.
Instead of having the large Target model generate tokens one by one, the generation process splits into a high-speed feedback loop:
[ Step 1: Draft Phase ]
Draft Model (1B) rapidly guesses K candidate tokens:
Prompt ──> [token_1] ──> [token_2] ──> [token_3] ──> [token_4] ──> [token_5]
(Extremely fast: 180+ tokens/sec)
[ Step 2: Verification Phase ]
Target Model (8B) verifies ALL 5 tokens simultaneously in ONE forward pass:
[token_1, token_2, token_3, token_4, token_5] ──> Parallel Tensor Evaluation
(Memory transferred ONCE, idle compute cores fully engaged!)
[ Step 3: Acceptance & Correction ]
Tokens accepted: [token_1: Accept] [token_2: Accept] [token_3: Accept] [token_4: Reject]
Output: 3 tokens accepted + 1 corrected token sampled from Target Model.
Net yield: 4 tokens generated in the time of a single forward pass!
Why Verification Is Nearly Free
Evaluating 5 tokens in a single parallel batch (prefill mode) takes almost the exact same amount of time as evaluating 1 single token!
Why? Because reading the 4.6 GB model weights from VRAM takes the same amount of time whether you multiply those weights against 1 token vector or 5 token vectors in parallel. By bundling multiple tokens into a single memory transfer, you finally wake up the idle Tensor cores and put them to work.
The Mathematical Guarantee: Zero Quality Loss
When people first hear about speculative decoding, their immediate suspicion is that the smaller, dumber draft model will degrade output quality.
It does not. Speculative decoding is provably lossless.
The verification step uses a specialized statistical algorithm called Speculative Rejection Sampling.
When the draft model proposes a candidate token, the target model evaluates its probability against its own internal distribution:
- If the target model agrees that the token is probable, the token is accepted unconditionally.
- If the target model assigns a lower probability, the token is accepted with probability proportional to the ratio of their distributions.
- If rejected, the runtime discards all subsequent draft tokens and immediately samples a replacement token directly from the target model's corrected probability distribution.
Because of this rejection sampling theorem (first proven by Leviathan et al. in 2022), the mathematical distribution of the output tokens is identical to running the target model standalone. Every sentence, every reasoning chain, and every comma generated is bit-for-bit what the larger model would have written.
Consumer VRAM Economics: Fitting on 8GB & 12GB GPUs
Can you actually fit both models into consumer VRAM?
The combination of modern compact draft models (such as Meta's Llama 3.2 1B paired with Llama 3.1 8B, or Qwen 2.5 0.5B paired with Qwen 2.5 7B/14B) makes speculative decoding exceptionally accessible for consumer setups:
| Model Component | Model Architecture | Quantization | VRAM Allocation |
|---|---|---|---|
| Target Model | Llama-3.1-8B-Instruct | Q4_K_M | ~4.92 GB |
| Draft Model | Llama-3.2-1B-Instruct | Q4_K_M | ~0.85 GB |
| KV Cache (Combined) | 4,096 Context | FP16 / Q8 | ~0.65 GB |
| Total VRAM Footprint | Dual-Model Rig | — | ~6.42 GB |
As the table shows, an 8B + 1B speculative pair requires under 6.5 GB of VRAM.
That means laptops with 8GB GPUs—like our Dell G15 Workhorse or Alienware 16X—can run speculative decoding with room to spare. On 12GB desktops like our Neon Future (RTX 5070) or Predator Orion, you can even upgrade to a 14B target model paired with a 1.5B draft model!
The Acceptance Rate: When Does It Fly, and When Does It Fumble?
The speedup factor of speculative decoding depends directly on one variable: the Acceptance Rate—the percentage of draft tokens that the target model approves.
- High Acceptance (75% to 85%): Repetitive text, boilerplate code (HTML, CSS, SQL), summarization, and predictable prose. In these domains, the draft model guesses accurately, and generation velocity easily hits 1.8x to 2.3x of baseline speed.
- Low Acceptance (40% to 50%): Highly abstract mathematical proofs, obscure cipher puzzles, or extreme creative writing where word choice is unpredictable. In these domains, the verifier rejects more tokens, and the speedup drops to 1.1x to 1.3x.
Crucially, speculative decoding almost never runs slower than baseline, provided the draft model is sufficiently small (under 15% of target model size) and both models reside fully in VRAM.
How to Enable Speculative Decoding Today
Both of the most popular open-source inference backends support speculative decoding natively out of the box.
1. In llama.cpp
llama.cpp provides native speculative sampling via the --model-draft (or -md) flag:
./llama-cli \
-m models/Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-md models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
-ngl 99 -ngld 99 \
--draft-max 5 \
-p "Explain the difference between synchronous and asynchronous I/O:" \
-n 512
Note: -ngl 99 offloads all target layers to GPU; -ngld 99 offloads all draft layers to GPU.
2. In vLLM
If you are running an OpenAI-compatible API server locally using vLLM, enable speculative decoding with:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--speculative-model meta-llama/Llama-3.2-1B-Instruct \
--num-speculative-tokens 5 \
--gpu-memory-utilization 0.90
Conclusion: The Smarter Way to Scale Local AI
For years, the hardware conversation in local AI has focused on brute force: buying wider memory buses, higher-wattage power supplies, and thicker graphics cards.
Speculative decoding proves that software architecture and statistical ingenuity can deliver hardware-level breakthroughs for free. By letting a lightweight draft model absorb the latency of memory bandwidth while allowing the primary model to flex parallel GPU compute, we can double generation speeds on existing hardware without sacrificing intelligence.
If you have an 8GB or 12GB GPU sitting on your desk, stop letting your Tensor cores idle—pair your models, enable speculative decoding, and experience the speedup firsthand.
Browse empirical GPU and CPU performance benchmarks across our active lab fleet on our Fleet Dashboard, or monitor consumer GPU memory availability on our Hardware Tracker.