vinpix/Bonsai-8B-llama.cpp

Ternary-Bonsai-8B for llama.cpp

GGUF builds of PrismML's Ternary-Bonsai-8B for llama.cpp, covering both deployment targets: a native TQ2_0 build for CPU, and a ternary-aware zero-exact Q2_K build for GPU and every k-quant runtime — at matching perplexity.

Ternary-Bonsai is an end-to-end 1.58-bit model (weights in {-1, 0, +1}, trained with ternary QAT, no higher-precision escape hatches). It punches far above its size class, but on llama.cpp it runs into a practical wall: the tight ternary format has no GPU kernel. This repo provides both halves of the solution.

Files

File	Format	BPW	Size	Backends	PPL (wikitext-2, 20x512)
Bonsai-8B-TQ2_0.gguf	native ternary	2.06	2.50 GB	CPU only	12.311 +/- 0.517
Bonsai-8B-Q2_KT.gguf	ternary-aware Q2_K	2.911	2.99 GB	CPU - CUDA - Metal - Vulkan - ROCm - SYCL	12.299 +/- 0.516

Both decode to the same learned weights. They differ only in storage container, and therefore in which backends can run them.

TL;DR — which one? CPU box -> TQ2_0 (fewest bytes streamed, smallest RAM, fastest generation on CPU). GPU offload, or any runtime that only speaks k-quants -> Q2_KT.

---

Why two files

TQ2_0 — the native CPU format

TQ2_0 is the tightest faithful ternary layout in ggml (~2.06 BPW). It is the right choice on CPU and it is what PrismML ships. Its one hard limit:

TQ2_0 has no GPU kernel. In current llama.cpp it implements a vec_dot for CPU only — no CUDA, Metal, or Vulkan path. -ngl silently keeps the ternary tensors on the host.

Q2_KT — the GPU-portable format

Q2_K is one of the most broadly supported quant types in the ecosystem — hand-written kernels for CUDA, Metal, Vulkan, ROCm, SYCL. Re-hosting the ternary weights inside a Q2_K container makes Bonsai portable everywhere Q2_K runs.

The catch: you cannot just run llama-quantize ... Q2_K on a ternary model. Doing so is actively harmful — for a reason worth explaining.

---

The trap: naively quantizing a ternary model to Q2_K

Q2_K dequantizes each weight as

w = d * (scale & 0xF) * q  -  dmin * (scale >> 4),   q in {0, 1, 2, 3}

The stock Q2_K quantizer (make_qkx2_quants) is an L2-optimal grid fitter for continuous FP16 distributions. Handed a ternary tensor whose values are exactly {-s, 0, +s}, it fits a 4-level grid over 3-level data and places the levels so the ternary zero lands between grid points — at q ~ 1.5, rounding to an offset of +/-s/3.

Measured directly on Bonsai-8B (against the ternary weights, read in ggml's canonical qs order):

Encoding	Non-zero weights (sign + magnitude)	Zero reconstruction
Stock Q2_K (--pure)	preserved (100 % signs, magnitude err ~1e-5)	-0.0095 — shifted to ~`±s/3`
Q2_KT (this repo)	preserved (100 % signs)	0.000000 — exact

The stock quantizer reproduces the non-zero ternary levels faithfully; its one failure on ternary data is the zero. Because a ternary model is ~1/3 zeros, sending every zero to ±s/3 injects error across a third of all weights — enough that a --pure Q2_K degrades badly (it loops single tokens). The stock pipeline masks this by promoting sensitive tensors (attn_output, ffn_down, attn_v) to Q3_K/Q4_K, inflating the file to ~3.18 BPW and only partially repairing the damage. That promotion is a workaround for the zero-error, not a fix.

Q2_KT fixes the encoding at the source.

---

How Q2_KT is built (ternary-aware, zero-exact)

The ternary structure admits an exact Q2_K encoding with no residual. For a group of magnitude s:

scale nibble  sc = 15
min   nibble  m  = 15          (min-nibble == scale-nibble)
d = dmin = s / 15
q = t + 1                      t in {-1, 0, +1}  ->  q in {0, 1, 2}

Substituting into the dequant:

w = d*15*q - dmin*15 = d*15*(q - 1) = s*(q - 1) = s*t     <- exact ternary

The -1 bias falls out of the min term for free; the ternary zero (t = 0 -> q = 1) reconstructs to exactly 0; every non-zero to exactly +/-s. The only error is the FP16 storage of the single per-group scale — which TQ2_0 also carries. Hence sign accuracy 100 % and PPL parity.

The qs nibbles are packed with the exact ggml Q2_K interleave (32-byte groups, stride-32 crumbs across 128-element spans), so the file is bit-compatible with the reference dequantize_row_q2_K and the SIMD vec_dot_q2_K_q8_K in the forward pass. Output projection and token embedding are kept at Q4_0 (not ternary in the source).

Layout note. TQ2_0 and Q2_K qs streams share the same stride-32 / 128-span interleave. Matching ggml's canonical order is the whole game: an encoding that is self-consistent with a hand-rolled unpacker but disagrees with the canonical order will pass every fidelity check against itself and still emit garbage in the forward pass, because dequantize_row and vec_dot both assume the canonical order. This build matches it.

---

Usage

CPU (use TQ2_0)

llama-cli -m Bonsai-8B-TQ2_0.gguf -p "The capital of France is" -n 64

Smallest RAM footprint and the fastest generation of the two on CPU (it streams the fewest bytes per token). This is the recommended file for CPU-only inference.

GPU offload (use Q2_KT)

llama-cli -m Bonsai-8B-Q2_KT.gguf -ngl 99 -p "The capital of France is" -n 64

Verified running fully offloaded on an AMD Radeon iGPU via the Vulkan RADV backend. Because Q2_K has real GPU kernels, -ngl 99 actually places the ternary body on the device — unlike TQ2_0, which would stay on the host. The same file runs on CUDA, Metal, ROCm and SYCL.

Faster prompt processing: Q2_KT on ik_llama.cpp

For prompt-heavy workloads (long contexts, RAG, batch processing), ik_llama.cpp — a performance-focused llama.cpp fork — runs Q2_KT noticeably faster in prompt processing thanks to its iqk_mul_mat kernels, which unpack the quantized weights once per batch and reuse them across the batch rows. It needs no special file: point it at Q2_KT and enable runtime repack with -rtr.

# build the fork
git clone https://github.com/ikawrakow/ik_llama.cpp && cd ik_llama.cpp
cmake -B build -DGGML_NATIVE=ON && cmake --build build --config Release -j

# run Q2_KT with runtime repack
./build/bin/llama-cli   -m Bonsai-8B-Q2_KT.gguf -rtr 1 -ngl 0 -p "The capital of France is" -n 64
./build/bin/llama-bench -m Bonsai-8B-Q2_KT.gguf -rtr 1 -p 512 -n 128   # to benchmark

In our tests this gave a solid double-digit-percent gain in prompt-processing throughput over stock llama.cpp on the same machine, with generation speed essentially unchanged — worth it when you feed long prompts. Two caveats: ik_llama.cpp does not implement the mainline TQ2_0 type, so this path is for Q2_KT only; and it is a separate fork, so treat it as an optional speed-up rather than a drop-in replacement for your stock toolchain.

Any GGUF loader / llama.cpp server

Q2_KT loads as an ordinary Q2_K model — no custom code, no patched runtime. Every existing loader already understands the container.

---

Quality

Perplexity on wikitext-2-raw (test split, 20 chunks of 512 tokens, identical harness):

Model	Format	BPW	PPL (lower is better)
Bonsai-8B-TQ2_0	native ternary	2.06	12.311 +/- 0.517
Bonsai-8B-Q2_KT	ternary-aware Q2_K	2.911	12.299 +/- 0.516

Q2_KT is perplexity-neutral vs. the native ternary build (delta within the +/-0.5 statistical error). You pay ~0.85 BPW over TQ2_0 for universal backend support; you do not pay for it in quality — expected, since the encoding is lossless on the ternary levels.

For task benchmarks (MMLU-Redux, GSM8K, HumanEval+, IFEval, ...), see PrismML's numbers for the base model — these files change the container, not the learned weights.

---

Technical notes & reproducibility

• Q2_KT container: 252 ternary body tensors -> pure Q2_K (zero-exact); output + token_embd -> Q4_0; 145 norms -> F32.
• No tensor promotion. Every ternary tensor is pure Q2_K; the zero-exact encoding makes the Q3_K/Q4_K promotion unnecessary — that is why Q2_KT is 2.91 BPW, not 3.18.
• Bit-exactness: validated against ggml's reference dequantize_row_q2_K (100 % sign accuracy, max abs error 1e-5 from the FP16 scale) and end-to-end against the forward vec_dot (coherent generation, PPL parity).
• On sub-2-bit formats: IQ1_BN (1.62 BPW) and IQ2_BN (2.0 BPW) were also evaluated via ik_llama.cpp. Both generate coherently — the per-group ternary scales survive quantization — but on CPUs without dedicated low-bit instructions their heavier per-weight decode makes generation slower than TQ2_0 despite the smaller footprint: the bottleneck shifts from memory bandwidth to decode compute. TQ2_0 remains the CPU generation-speed choice because it is simultaneously byte-light and decode-cheap.
• Architecture: Qwen3, 36 layers, hidden 4096, FFN 12288, GQA 32/8, full 65536-token context preserved in metadata.
• Reproduce PPL: llama-perplexity -m <file> -f wiki.test.raw --chunks 20.

A note for the llama.cpp community

The +/-s/3 zero-error affects any end-to-end ternary model (Bonsai, BitNet, TriLM, ...) run through the stock Q2_K quantizer. A ternary-aware branch in quantize_row_q2_K — detect a block with <=3 symmetric levels, emit d = dmin = s/15, sc = m, q = t+1 — would let the official pipeline produce zero-exact Q2_K for the whole class, without the promotion workaround. Q2_KT is a concrete demonstration that the encoding is exact and backend-portable. Contributions upstream welcome.

---

Attribution & license

• Base model: Ternary-Bonsai-8B by PrismML (Babak Hassibi et al., Caltech). Ternary QAT, Qwen3 architecture, end-to-end 1.58-bit.
• These files: GGUF re-encodes for llama.cpp. TQ2_0 is the native ternary layout; Q2_KT is a lossless-on-ternary Q2_K re-encode for backend portability. Learned weights unchanged.
• License: Apache 2.0, inherited from the base model.

@techreport{ternarybonsai,
  title  = {Ternary Bonsai: 1.58-bit Language Models at 8B, 4B, and 1.7B Scale},
  author = {Prism ML},
  year   = {2026},
  month  = {April},
  url    = {https://prismml.com}
}

---

Community re-encodes, not affiliated with or endorsed by PrismML. Quality figures are measured on the specific files in this repository.