mudler / mudler/vllm.cpp

Matched-quantization leg: our engine on RadixArk/Qwen3.8-27B-NVFP4, the checkpoint pangoleen's SGLang result is measured on

Open
#2,761 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BENCH-QWEN38-27B-SOTA

Run our engine on RadixArk/Qwen3.8-27B-NVFP4, the checkpoint pangoleen/qwen3.8-27b-dgx-spark-dflash2 actually serves, so the quantization stops being a free variable in that comparison.

Spec: .agents/specs/bench-qwen38-27b-nvfp4-matched.md.

docs/benchmarks/qwen38-27b-exl3-gb10.md records 16.74 tok/s target-only on Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw. pangoleen records a 12.71 tok/s no-drafter median on RadixArk/Qwen3.8-27B-NVFP4 under SGLang, on the same class of box. Side by side that reads 1.32x in our favour, and it is not an engine result: read from both checkpoints' own safetensors headers, the weights a decode step sweeps once per token are 11.874 GB here and 17.608 GB there. Byte-normalized, 16.74 x 11.874 = 199 GB/s against 12.71 x 17.608 = 224 GB/s, so the comparison reverses sign to 0.888x, and against the 231 GB/s their box measures their target forward is at 97% of the bus while ours is at 86%.

Two things follow. Their target-only decode has at most ~3% of headroom left in it, so "we beat them target-only on matched quantization" is not an outcome available to be won. And what is actually contested is the speculation multiplier and its mechanism metric: their accepted tokens per pass at budget 16 is 6.8-8.1, ours saturates at 6.24 from budget 12, on a drafter whose architecture is byte-identical to theirs.

Scope. The staged, sha256-pinned checkpoint pair, the device job at /workspace/nvfp4-sota/job1.sh, and a rung-by-rung comparison against their published ladder with spreads, the counting convention stated, and every unmatched axis named. Their own caution travels in reverse: boot-to-boot median absolute difference 4.3%, worst 8.1%, so a difference smaller than 4.3% against one of their rungs is not a difference.

Already established, hermetically, before any GPU time.

  • The config gates pass. RefusalForHfConfigRaw and RefusalForQuantizationConfig both answer "" on the real config.json and the real 2194 tensor names.
  • RadixArk and r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121 are the same weights; the committed 970-tensor shard-1 fixture is a subset with zero dtype or shape disagreements.
  • unsloth/Qwen3.8-27B-NVFP4, which sits on the NAS, is not loadable: the same probe answers with a 233-module refusal naming per-output-channel weight_scale and dynamic per-token activations.

Blocked on, and tracked separately: #2758 (no NVFP4 arm for the DFlash2 draft weights, so their drafter cannot be loaded at all), #2759 (no harness can honour EOS with a chat template, so their protocol cannot be matched), #2760 (their target's declared W4A4 runs our W4A16 arm silently).

Status. The device job is queued on dgx:gpu0 at position 10 behind a 10-hour job. No number is measured.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read .agents/specs/bench-qwen38-27b-nvfp4-matched.md and docs/benchmarks/qwen38-27b-exl3-gb10.md, then inspect /workspace/nvfp4-sota/job1.sh. Run the staged, pinned checkpoint comparison when dgx:gpu0 is available, recording each ladder rung, spreads, counting convention, and unmatched axes. Done means a completed byte-normalized and speculation comparison with no unsupported engine result claims.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.