What Actually Fits on Dual RTX 3090s: Qwen 27B and the KV Cache Math

Two RTX 3090s give you 48GB for around the price of one current-generation card, and a 27B model with a six-figure context window is exactly the workload people buy them for. The question is what that setup genuinely delivers.

The short answer: the context claims are real and better than advertised. The speed claims usually aren't. This guide shows the arithmetic for both so you can check any number you see, including the ones here.

The thing that makes this work isn't quantization

Qwen3.5-27B is routinely described as a dense 27B model. The FFN is dense — it isn't a mixture of experts — but the attention stack is not what "dense transformer" implies:

Verify it yourself:

curl -sL https://huggingface.co/Qwen/Qwen3.5-27B/raw/main/config.json \
  | python3 -c "import json,sys; c=json.load(sys.stdin)['text_config']; \
    lt=c['layer_types']; print(len(lt),'layers |',lt.count('full_attention'),'full |',lt.count('linear_attention'),'linear')"
# 64 layers | 16 full | 48 linear

Linear-attention layers carry a fixed-size recurrent state instead of a KV cache that grows with context. Three quarters of this model's layers cost the same at token 200,000 as at token 1. That is the entire reason long context fits on consumer hardware, and it's an architectural property — no amount of quantization would have got you there.

The KV cache arithmetic

Only the 16 full-attention layers cache:

KV elements per token = 2 (K and V) × 4 kv_heads × 256 head_dim × 16 full-attn layers
                      = 32,768 elements  →  64 KiB/token at FP16

Against a Q4_K_M weight file of 16.74 GB (measured, unsloth/Qwen3.5-27B-GGUF), on 48GB total with roughly 44–45GB usable after CUDA context and activations:

170K on dual 3090s is not a stretch — it's comfortable, at full FP16 KV, with no quantization of the cache required. The frequently-quoted 170K figure undersells the hardware; the architecture's native limit is 262K and that fits too.

Now the counterfactual that shows why this matters. Had all 64 layers been full attention:

2 × 4 × 256 × 64 × 2 bytes × 170,000 = 44.6 GB   ← KV cache alone

Plus 16.74 GB of weights. It would not fit, on any KV quantization scheme, at any context near that length. The hybrid architecture isn't a footnote to the story — it is the story.

What decode speed to actually expect

Single-stream decode is bound by weight bandwidth: every token requires streaming the whole model. A 3090 has ~936 GB/s.

Streamed bytes ≈ 16.74 GB weights − ~1.3 GB input embedding table (gathered, not streamed) ≈ 15.7 GB

One 3090, theoretical:     936 / 15.7  =  59.6 t/s
Two 3090s, TP=2, theory:   936 / 7.85  = 119.2 t/s

So a claim of 100+ t/s single-stream is asking for 84% of the two-card theoretical ceiling — essentially perfect bandwidth efficiency, before paying anything for the interconnect. Real measurements on this exact model and hardware:

That 41.4 is about 70% of the single-card roofline, which is the expected result: llama.cpp splits by layer rather than tensor-parallel, so at batch=1 only one card's bandwidth is working at a time.

Why the second card helps less than you'd hope

Tensor parallelism splits each layer across both GPUs, which halves the bytes per card but adds two all-reduce collectives per layer — 128 round trips per token at 64 layers. Each carries only ~10 KB, so this is latency-bound, not bandwidth-bound, and on PCIe-connected Ampere the latency largely eats the bandwidth win at batch=1.

The 3090 is the last consumer NVIDIA card with an NVLink bridge connector (~112.5 GB/s versus ~31.5 GB/s for PCIe 4.0 x8), so it can help here — but published dual-3090 recipes run PCIe-only and explicitly disable NVLink-specific NCCL paths, so don't treat a bridge as mandatory.

Tensor parallelism pays off properly at batch, where the weight read is amortised across concurrent sequences. Prefill and throughput are where two cards shine:

If you see ~1,500 t/s prefill quoted for this setup, that's conservative rather than optimistic.

The way to actually reach 100+ t/s

Speculative decoding — MTP, or the newer DFlash/DFlash2 — drafts several tokens and validates them in one weight pass, breaking the bandwidth roofline that caps ordinary decode. Published single-3090 results for the same 27B architecture show 46 t/s without it and 121–133 t/s with it.

Two things worth knowing before you plan on this:

1. Acceptance rate is content-dependent. Reported dual-3090 numbers split roughly 71 t/s on prose versus 89 t/s on code with MTP, and 78 versus 128 with DFlash. Your prose workload will not see the code number.
2. It was not a working option in early 2026. The Gated DeltaNet kernels were crashing — vLLM #34948 (CUDA Illegal Memory Access in GDN Kernel, February, still open) and #36010 (Qwen3.5-27B Batch Inference very slow / not working, filed 4 March 2026). Speed claims from that window predate a usable speculative decoding path.

The long-context trap on llama.cpp

If you are chasing a six-figure context specifically, this one will cost you a day:

The VRAM fits. The engine doesn't, yet. Test your actual target context with a retrieval check before building on it — a fact planted at the start of a 170K prompt, retrieved at the end. Don't infer working long context from a successful model load.

What changed since early 2026

Anything written about this setup before roughly April 2026 is describing a different landscape:

• Speculative decoding matured. The single biggest change, and the only route to triple-digit single-stream decode.
• TurboQuant KV quantization didn't exist. First requested in vLLM #38201 on 26 March 2026, with hybrid-model support landing in April. Its 3-bit mode substantially increases how many concurrent long-context streams fit, at roughly 25% per-stream throughput cost.
• TP=2 on int4 needed a fix. Marlin sub-tile padding (vLLM #40361); without it, loading crashes with GPTQ_MARLIN_MIN_THREAD_N (64) > out_features.
• Context parallelism arrived, and is still rough — --decode-context-parallel-size exists but carries open output-drift and prefix-caching issues.
• The model is two generations old. Qwen3.6-27B and Qwen3.8-27B are architecture-compatible successors.

Choosing a configuration

Start from what you're optimising for, not from someone else's headline:

Then measure, on your own prompts:

vllm bench serve --model <model> --dataset-name random \
  --random-input-len 8192 --random-output-len 512 --max-concurrency 1

Single-stream and batch numbers answer different questions, and prompt length changes both. A benchmark that doesn't state concurrency and input length isn't telling you enough to act on.

Next steps

• Model card and config: Qwen/Qwen3.5-27B — and check the current 27B before committing
• Cross-platform measurements with stated methodology: BAEM1N/llm-bench
• Long-context correctness issue: llama.cpp#27756
• The post that prompted this guide: Qwen 3.5 27B Achieves 100+ Tokens/s Decode on Dual RTX 3090s