bw24-bench — benchmark artifacts for bw24

Reproduction artifacts for the performance claims in github.com/avifenesh/bw24 — a from-scratch LLM inference engine for RTX 5090 Laptop (sm_120a). The README there compares bw24 against llama.cpp with specific speculative-decoding configs and vocab-trimmed draft heads. Those trims and prompts are not standard files you can download elsewhere, so they are published here: if the claim depends on a specific artifact, the artifact must be public.

What is in this repo

Trimmed MTP draft-head GGUFs (Qwen3.6-27B)

Three FR-Spec vocab-trimmed variants of the Qwen3.6-27B MTP draft head (Q4_K_M). All three are the same model-trained head (original Qwen3.6 release) with a pre-defined limited draft vocabulary: a d2t ranked-token subset (32768 tokens) over the full Qwen vocab, built from token-frequency analysis. Trimming shrinks the draft lm_head reads ~5x; it does not retrain anything.

File	Ranking	Use
mtp-Qwen3.6-27B-Q4_K_M-frspec32768.gguf	generic frequency-ranked cover	best for medium-code / long-agentic prompts (p2/p3)
mtp-Qwen3.6-27B-Q4_K_M-frspec-code75-32768.gguf	code-skewed (75% code corpus)	best for short-code prompts (p1) only
mtp-Qwen3.6-27B-Q4_K_M-frspec-balanced32768.gguf	balanced code/prose mix	superseded by the two above; kept for the experiment record

Measured lesson (2026-07-07, in the tune-data corpus): trim rankings are class-dependent and head-specific. code75 wins only on short-code content; the generic ranking wins everywhere else; none of these rankings transfer to a differently-trained head (e.g. the NVIDIA-finetuned checkpoint's own head) — re-derive the ranking from the target head's draft distribution before trimming.

Each trim pairs with bw24 via BW24_FRSPEC_TRIM=<path-to-trim.gguf>.

9B and safetensors-model trims (trims/)

9B-native trim (frspec-9b-32768.gguf): The Qwen3.5-9B has a different vocabulary (248320 tokens vs the 27B/35B's 151936), so the trims above cannot transfer to it. This d2t was built with bw24's frspec_rank tool (in-repo): tokenize a local ~2.6M-token code/docs corpus with the 9B's own GGUF tokenizer, rank ids by frequency, keep the top 32768. Measured effect on the 9B raw-prompt spec board (2026-07-08): p1 199→243, p2 166→195, p3 142→162 tok/s (+22/+17/+14%), self-consistency PASS. The trim mechanism byte-gathers rows from the running model's own lm_head — the file carries only the ranked id list (131 KB).

Safetensors-path trims (v0.8.0):

• frspec-9bst-modelopt-32768.gguf: FR-Spec trim for the Qwen3.5-9B modelopt NVFP4 safetensors checkpoint. Same ranking method as the GGUF 9B trim, but derived from the safetensors model's own MTP head distribution.
• frspec-corpus-32768.gguf: FR-Spec trim for the NVIDIA official Qwen3.6-27B NVFP4 safetensors checkpoint. Corpus-ranked (same ~2.6M-token corpus as the 9B trims).

A second cross-model lesson (2026-07-08): between models that DO share a tokenizer (27B/35B), the generic ranking transfers unmodified — the 35B uses the 27B's frspec32768 file directly (spec p2 174.5→187.1 from the trim alone).

Bench prompts (prompts/)

The exact prompts behind the README numbers, plus the long-context set:

File	Size	Class
p1-code-short.txt	146 B (28 tok)	short code
p2-code-medium.txt	6.3 KB (~1.8k tok)	medium code
p3-agentic-long.txt	23 KB (~6.3k tok)	long agentic
p3-agentic-long-v2.txt	23 KB (~6.3k tok)	long agentic (experimental v2)
p4-16k.txt / p5-32k.txt / p6-64k.txt	79/135/247 KB	long-context prefill set

Note on p3-agentic-long: A text audit found the greedy continuations on this prompt partially degenerate into repetition on both engines (bw24 and llama.cpp) symmetrically. The ratios between engines remain valid, but absolute tok/s values overstate real agentic throughput. See rig5090.jsonl tag p3-degeneration-audit in the main repo. A replacement prompt is pending; p3-agentic-long-v2.txt is the experimental candidate.

Exact configs (CONFIGS.md)

Complete serve configurations for all v0.8.0 board rows:

• bw24 safetensors rows (Qwen3.5-9B ST and Qwen3.6-27B ST): plain and speculative configs with exact flags, K values, pmin thresholds, and trim paths per content class (p1/p2/p3)
• llama.cpp comparison configs: exact build info (commit, CUDA version, compile flags) and serve commands (llama-bench and llama-server) used to generate the comparison numbers

See CONFIGS.md for the full reproducibility commands.

Numeric config note (2026-07-08)

BW24_FA_V2 (tile-batched online-softmax FA decode) became the engine default on 2026-07-08 — a one-time numeric-config change, gated by the full battery (kernel-check, argmax MATCH, K=1..8 spec self-consistency, graph bit-identity) on all three models. Plain-decode boards measured with it: d512 9B 132.7 / 27B 47.7 / 35B 173.4 vs llama 124.6 / 43.5 / 170.5; d6257 124.5 / 44.9 / 158.5 vs 119.6 / 42.0 / 159.9. Reproducing pre-2026-07-08 numbers requires BW24_FA_V2=0.

Prefill numbers use BW24_PP_ONLY=1 BW24_PP_REPS=3 (pp-only reps, median). Inline prefill timing inside a full generate run understates pp by ~20% — it carries the correctness-gate overhead; same-day both-protocol measurement on the 27B read 1055 inline vs 1297 pp-only on identical code.

Exact bench protocol

• 256 generated tokens, temperature 0, N=3 runs, report the median.
• Plain runs need no flags — the fast path is the default. The f32 oracle path (correctness reference): BW24_FAST=0.
• Speculative runs set: BW24_SPEC_K=<K> BW24_SPEC_PMIN=<pmin> BW24_SPEC_HPOST=1 BW24_FRSPEC_TRIM=<trim.gguf> (+BW24_SPEC_PMIN0=1 for the 35B)
• Runner: research/e2e/run-e2e.sh in the bw24 repo (same prompts fed to both engines).
• Speculative output is gated bit-exact: K=1..8 self-consistency check pins it token-identical to plain greedy decode.

Per-class configs behind the README spec board (2026-07-08, raw-prompt protocol, both engines full power)

Qwen3.6-27B (108 / 91 / 79.5 vs llama 86.4 / 89.9 / 73.2 = 1.25x / 1.01x / 1.09x):

Prompt	K	pmin	Trim	bw24 tok/s	llama.cpp tok/s
p1 short code	3	0.15	generic frspec32768	108.4	86.4
p2 medium code	3	0.3	generic frspec32768	90.9	89.9
p3 agentic long	3	0.3	generic frspec32768	79.5	73.2

Qwen3.5-9B (243 / 195 / 162 vs llama 186 / 158 / 155 = 1.31x / 1.23x / 1.04x): K=3, pmin 0.15, BW24_SPEC_HPOST=1, trim trims/frspec-9b-32768.gguf (the 9B-native ranking — the 27B trims cannot transfer, different vocab).

Qwen3.6-35B-A3B (202.8 / 200.5 / 184.8 vs llama self-MTP 215 / 208 / 202 — post FA_V2 default, 2026-07-08 evening): K=2, pmin 0.4, BW24_SPEC_PMIN0=1 (zero-draft rounds), trim = the 27B frspec32768 file (transfers unmodified — same tokenizer; the d2t gather reads the 35B's own lm_head). Plus BW24_MOE_CACHE=1.

Config laws from the corpus: K optimum is (content-class, protocol) dependent — chat-templated prompts shift both K and the trim choice (chat short-code runs K=7 + code75 at 122 tok/s on the 27B; raw continuation runs K=3 + generic). Zero-draft rounds pay below ~75% base acceptance and hurt above ~90%. Trim before gating — ungated pmin sweeps measure flat without a cheap probe.

llama.cpp comparison config

llama.cpp built on the same machine at commit 047bfa508 (includes the FR-Spec d2t draft-vocab-trim patch for MTP), run at its best serve config:

llama-server -m Qwen3.6-27B-NVFP4-Q4_K_M-mtp.gguf \
  --model-draft mtp-Qwen3.6-27B-Q4_K_M.gguf --spec-type draft-mtp \
  --spec-draft-n-max 3 --spec-draft-p-min 0.1 -ngld 999 \
  --ctx-size 16384 -ngl 999 -fa on --cache-type-k q8_0 --cache-type-v q5_1

Protocol note from the corpus: sequential cross-session A/B numbers lie by up to 10% (clock behavior differs when an engine runs alone/cold) — interleave both engines within the same minute, N>=3 pairs, both orders.

Full experiment corpus

Every tuning experiment (positive and negative) is recorded as JSONL at github.com/avifenesh/bw24 — research/tune-data/. The rows named qwen36-27b-nvfp4-q4km and qwen35-9b-nvfp4 / qwen36-35b-a3b-iq4xs dated 2026-07-07/08 are the ones behind the table above.

Format decision (2026-07-10)

GGUF NVFP4 is bw24's primary, board-published format (chosen on long-context serving reliability). The ST (safetensors) configs below remain published as best-effort support.