llama2 gguf with 2bit quantisation only needs ~5gb vram. 8bits need >9gb. Anything inbetween is possible. There are even 1.5bit and even 1bit options (not gguf AFAIK). Generally fewer bits means worse results though.
Yeah, I usually take the 6bit quants, didn’t know the difference is that big. That’s probably why tho. Unfortunately, almost all Llama3 models are either 8B or 70B, so there isn’t really anything in between but I find Llama3 models to be noticeably better than Llama2 models, otherwise I would have tried bigger models with lower quants.
That’s weird, maybe I actually am doing something wrong. Is it because I’m using GGUF models maybe?
llama2 gguf with 2bit quantisation only needs ~5gb vram. 8bits need >9gb. Anything inbetween is possible. There are even 1.5bit and even 1bit options (not gguf AFAIK). Generally fewer bits means worse results though.
Yeah, I usually take the 6bit quants, didn’t know the difference is that big. That’s probably why tho. Unfortunately, almost all Llama3 models are either 8B or 70B, so there isn’t really anything in between but I find Llama3 models to be noticeably better than Llama2 models, otherwise I would have tried bigger models with lower quants.