I mean, if you’re really patient you can even run with CPU only as long as you have enough RAM… My ~decade old computer with DDR3 RAM gets 14 tok/s prompt processing and ~5 tok/s decode on MoE models (e.g. a Qwen 3.6 35B-A3B variant at Q6_K – my current default model). That pp tok/s is painfully slow… but it still works!
Same computer, same model but using a discrete GPU that can fit all the weights on the card – ~900 tok/s prompt processing, ~70 tok/s decode.
I don’t think you can run 27B on 16GB without degrading its quality. You’d have to drop to lower quantization.
35B A3B runs well on 8 and 16GB GPUs.
Yeah, down to 4bit, which works OK, but 35BA3B is much more performant. Sometimes you want the monolithic quality and damn the speed…
Everything runs well if one is patient enough 😂
But yeah, with a smaller context and carefully offloading some of the layers to the CPU, it’ll run just not very fast. And definitely using Q4 quants.
I mean, if you’re really patient you can even run with CPU only as long as you have enough RAM… My ~decade old computer with DDR3 RAM gets 14 tok/s prompt processing and ~5 tok/s decode on MoE models (e.g. a Qwen 3.6 35B-A3B variant at Q6_K – my current default model). That pp tok/s is painfully slow… but it still works!
Same computer, same model but using a discrete GPU that can fit all the weights on the card – ~900 tok/s prompt processing, ~70 tok/s decode.
IQ3s perform very well these days.