Homelabber uses second-gen Optane memory sticks to run a 2.8-trillion-parameter AI model on CPU alone

A homelab builder on Reddit has run Moonshot AI’s 2.8-trillion-parameter Kimi K3 AI model on discontinued second-gen Optane PMem 200 memory, using a 40-core 3rd-gen Xeon Scalable CPU and not a GPU for inference. The build runs hot, and the builder’s advice is that you “treat active DIMM cooling as mandatory,” with dual 12W blower fans aimed at the modules to get temperatures under control. The project, a follow-up to a separate user’s first-gen Optane build earlier this year, runs at a sluggish 0.85 tokens per second even with 1-bit quantization.

Kimi K3 is absolutely not built for home use, in our estimation, with Moonshot AI recommending data center inference engines to run it, and even the smallest community quantization needs at least 610GB of RAM. Using secondhand Optane, instead, is one of the few ways to reach that capacity at home. …