llama-modchip Squeezes 284B DeepSeek V4-Flash Into 64GB of RAM, No GPU

What llama-modchip actually does

llama-modchip is an open-source set of ggml and llama.cpp kernel patches, quantization tools, and a command-line runner aimed at large mixture-of-experts models on ordinary system RAM. Its headline target is DeepSeek V4-Flash, a model with 284.33 billion total parameters, of which 13.17 billion are active for any given token.

The problem it attacks is simple. Even an aggressive 1-bit style build of the model, Unsloth's UD-IQ1_S, takes 76.87 GiB, which cannot fit on a Linux machine with about 62 GiB of usable memory. The model card says that forces weights to stream from the SSD on each token, with throughput collapsing as a result. llama-modchip's transcoded build comes to 56.56 GiB, and the author reports a peak of 57.61 GiB of resident memory at a 256K-token context with no swap.

How the 56.56 GiB build gets there

The model averages 1.71 bits per weight, but that figure hides a very uneven allocation. Most of the savings come from the 256 routed experts, which are held mainly at about 1.56 bits per weight, with sensitive layers kept slightly higher. Embeddings, shared experts, and other parts the author treats as the reasoning backbone stay at much higher precision.

The project's custom format, Fractal-64, runs at 3.75 bits per weight and only covers selected attention and indexer tensors. The catch is compatibility: these files use a new tensor type, so stock llama.cpp, Ollama, and LM Studio reject them, and you need the project's own binary. The tooling is Apache 2.0, while the published weights are marked MIT.

The speed numbers and what they mean

On a Ryzen 9 9955HX with 16 Zen 5 cores, 64GB of DDR5, and no GPU, the author reports prefill of 43.16 to 44.36 tokens per second and decode of 9.06 to 10.74 tokens per second. The 11.5 to 14.7 range in the project's summary depends on a speculative drafter, so it will vary with how predictable the text is.

Against the unpatched build, the gain is large, although the two documents give slightly different baselines for decode: 5.91 tokens per second in the README and 6.41 on the model card. The project also supports smaller models, including a 30B Nemotron variant, which is where its faster results appear.

Read the word lossless carefully

The README reports a logit cosine of 1.00000 and zero KL divergence for its "lossless" mode. That compares the patched runtime with the unpatched runtime on the same 1.71-bit checkpoint, and lossless here refers to keeping all six routed experts rather than skipping some. It does not show that the compressed model matches DeepSeek's full-precision weights, and the README we read offers no such comparison for the 284B build.

That gap matters because squeezing most weights to under two bits is exactly where quality loss would show up, especially in long reasoning or tool use. AI Weekly's write-up makes the same broader point: the figures come from one author and have not been independently reproduced. The repository only dates to 29 October, and its community leaderboard entries are self-submitted.

Why local AI users should watch it

If the results hold, the memory bar for running a frontier-class open model drops from a workstation with far more RAM to a well-specified desktop or laptop, with everything staying on your own machine. Ten tokens a second is slower than a good GPU, but it is usable for background agents and private document work. Until outside testers publish task benchmarks, treat this as a promising engineering result rather than a settled way to run DeepSeek at home.