Follow-up from “Dumbest Thing you have done distro-hopping?”.
Here’s mine - the laptop from which I’m typing right now has a broken touchpad that keeps jumping and clicking randomly, and does not work. Well, I can’t afford to fix it, but at the moment, I was so pissed off I punched the touchpad really hard, and the machine panicked with all the lights blinking. A few more revival abuses, and the machine was back to life, but since I was running a nixos-rebuild switch --upgrade in the background, I blew off my boot partition. I think I just broke the unbreakable distro.
If I didn’t mess around with LLMs I’d sell my Nvidia card in a heartbeat.
It performance that much better compared to a beefy CPU? I’ve been think about investing a beefy CPU to do compiles and run localAI.
The difference is huge between even the beefiest CPU and a decent GPU. My RTX 3080 does 7-8 tokens per second on the models I’ve tried while my Ryzen 9 5950x gets 1-2.
I’m not an expert with them nor have I done any optimization or benchmarking to find the real answer but it’s significant enough that I’d go for a GPU if I were building something dedicated for it
Your probably right but that involves a ton of proprietary software. CPU based systems decrease the amount of proprietary software required
Tbh, idgaf