Matched-quantization leg: our engine on RadixArk/Qwen3.8-27B-NVFP4, the checkpoint pangoleen's SGLang result is measured on
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BENCH-QWEN38-27B-SOTA
Run our engine on RadixArk/Qwen3.8-27B-NVFP4, the checkpoint pangoleen/qwen3.8-27b-dgx-spark-dflash2 actually serves, so the quantization stops being a free variable in that comparison.
Spec: .agents/specs/bench-qwen38-27b-nvfp4-matched.md.
docs/benchmarks/qwen38-27b-exl3-gb10.md records 16.74 tok/s target-only on Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw. pangoleen records a 12.71 tok/s no-drafter median on RadixArk/Qwen3.8-27B-NVFP4 under SGLang, on the same class of box. Side by side that reads 1.32x in our favour, and it is not an engine result: read from both checkpoints' own safetensors headers, the weights a decode step sweeps once per token are 11.874 GB here and 17.608 GB there. Byte-normalized, 16.74 x 11.874 = 199 GB/s against 12.71 x 17.608 = 224 GB/s, so the comparison reverses sign to 0.888x, and against the 231 GB/s their box measures their target forward is at 97% of the bus while ours is at 86%.
Two things follow. Their target-only decode has at most ~3% of headroom left in it, so "we beat them target-only on matched quantization" is not an outcome available to be won. And what is actually contested is the speculation multiplier and its mechanism metric: their accepted tokens per pass at budget 16 is 6.8-8.1, ours saturates at 6.24 from budget 12, on a drafter whose architecture is byte-identical to theirs.
Scope. The staged, sha256-pinned checkpoint pair, the device job at /workspace/nvfp4-sota/job1.sh, and a rung-by-rung comparison against their published ladder with spreads, the counting convention stated, and every unmatched axis named. Their own caution travels in reverse: boot-to-boot median absolute difference 4.3%, worst 8.1%, so a difference smaller than 4.3% against one of their rungs is not a difference.
Already established, hermetically, before any GPU time.
- The config gates pass.
RefusalForHfConfigRawandRefusalForQuantizationConfigboth answer""on the realconfig.jsonand the real 2194 tensor names. RadixArkandr0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121are the same weights; the committed 970-tensor shard-1 fixture is a subset with zero dtype or shape disagreements.unsloth/Qwen3.8-27B-NVFP4, which sits on the NAS, is not loadable: the same probe answers with a 233-module refusal naming per-output-channelweight_scaleand dynamic per-token activations.
Blocked on, and tracked separately: #2758 (no NVFP4 arm for the DFlash2 draft weights, so their drafter cannot be loaded at all), #2759 (no harness can honour EOS with a chat template, so their protocol cannot be matched), #2760 (their target's declared W4A4 runs our W4A16 arm silently).
Status. The device job is queued on dgx:gpu0 at position 10 behind a 10-hour job. No number is measured.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read .agents/specs/bench-qwen38-27b-nvfp4-matched.md and docs/benchmarks/qwen38-27b-exl3-gb10.md, then inspect /workspace/nvfp4-sota/job1.sh. Run the staged, pinned checkpoint comparison when dgx:gpu0 is available, recording each ladder rung, spreads, counting convention, and unmatched axes. Done means a completed byte-normalized and speculation comparison with no unsupported engine result claims.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100