Bonsai 2 27B's 5.95 GB GGUF Needs a Custom Runtime

Two compressed files make different trade-offs

PrismML published the Bonsai 2 27B model card on September 17. Its ternary weights, represented as negative one, zero, or positive one with group-wise scaling, reduce the 27B-class language model to roughly one ninth of its full-precision Qwen3.8 source. The dense PTQ1_0 packing produces the 5.95 GB file. A PQ2_0 alternative is 7.21 GB and takes less work to unpack on some hardware.

The smaller pack is not consistently faster. PrismML's own GPU tests show that the preferred format changes with the processor and whether it is processing a prompt or generating tokens.

Image input needs a separate 0.63 GB vision component. Both file sizes omit the operating system, runtime buffers, attention state, and room for the rest of an application. The advertised 262,144-token context is an architectural limit, not a sensible default on memory-constrained hardware. Longer prompts can delay the first visible output even if the weights themselves fit.

This is a meaningful change from Qwen's official 27B files: its FP8 repository totals 30.89 GB and its BF16 repository totals 55.59 GB. Bonsai's compression opens a different deployment route, but it is a separately produced artifact with a separate runtime requirement, not an update to those official files.

Stock llama.cpp does not run this GGUF

PrismML says the new PTQ1_0 and PQ2_0 formats need the ternary and hybrid-attention kernels in its llama.cpp fork. Stock llama.cpp rejects those types. The model card also warns that a Q2_0 file can load without a warning yet produce incorrect output because the matching activation transform is missing. A GGUF extension and a small file therefore do not imply compatibility with an existing local chat app.

The company publishes a tested setup and pinned binaries in its demo repository. That is a starting point for developers, not evidence that any Android model manager or WebGPU app supports this exact file today. Its reported throughput table covers NVIDIA GPUs and Apple M-series laptops. The additional M5 Max and M4 Pro figures were measured on an earlier build and await remeasurement on the current stack. There is no Android device, browser, or sustained thermal result in that table.

For a phone, an app would first need to integrate and qualify the modified runner, exact artifact, tokenizer, prompt settings, and any image path. Then it would need real-device checks for cold load, available RAM, prompt prefill, first visible output, generation speed, Stop and retry, and heat. CuriousLM has not qualified Bonsai 2 27B as a selectable model. The Android model-choice guide explains why a downloadable checkpoint is different from an app-supported option.

The quality figure is a vendor result

PrismML says its ternary model retains 98.2 percent of the full-precision model's aggregate score in its benchmark comparison. Its announcement and detailed model card report different benchmark sets and averages, so the percentage should not be read as an independently replicated, universal measure of quality. Coding, vision, instruction-following, and long-running tasks can lose different amounts after compression. TechCrunch independently confirmed the release and its size, but did not publish a controlled same-hardware quality or phone test.

For someone with a supported laptop or GPU, the smaller artifact is worth testing against the full model on actual tasks. For an Android user, the current decision is simpler: do not download 5.95 GB on the assumption that a general GGUF app will recognize it. Check the app's specific runtime support first, then leave memory headroom and measure the whole chat interaction, not just tokens per second.