mudler / mudler/vllm.cpp

A ModelOpt checkpoint declaring NVFP4 W4A4 runs our W4A16 arm silently: RadixArk/Qwen3.8-27B-NVFP4 ships input_scale on all 193

Open
#2,760 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: QUANT-QWEN38-27B-NVFP4-ARM

A ModelOpt checkpoint that declares quant_algo: "NVFP4" and ships an input_scale on every NVFP4 module runs this tree's W4A16 weight-only arm, silently, and says nothing.

RadixArk/Qwen3.8-27B-NVFP4 @ 554ebba9b5f1b79dc11246341960360e6ef05ef4 is the target pangoleen/qwen3.8-27b-dgx-spark-dflash2 serves under SGLang, and it is the artifact r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121 was derived from. Read from both config.json files at the pinned revisions, the two differ in exactly two ways that matter:

RadixArk @ 554ebba9 r0b0tlab @ 36f717a2
all 193 quantized_layers MLP entries {"quant_algo": "NVFP4", "group_size": 16} {"quant_algo": "W4A16_NVFP4", "group_size": 16}
config_groups.group_1.input_activations {dynamic: false, num_bits: 4, type: float, group_size: 16} null
<proj>.input_scale on an NVFP4 module 192 MLP + lm_head zero
kv_cache_scheme {dynamic: false, num_bits: 8, type: float} absent

Everything else agrees: the same 208 FP8 + 193 NVFP4 split over the same 401 module names, and shard 1's committed 970-tensor fixture tests/vllm/models/qwen38_27b_modelopt_mtp_s1_manifest.inc is a subset of RadixArk's 2194 tensors with zero dtype or shape disagreements. So r0b0tlab is RadixArk re-split, with the MTP shard separated, the input_scale tensors stripped and the algorithm relabelled weight-only.

What this tree does with the RadixArk spelling. Both whole-checkpoint refusal gates answer "" — compiled against 3047871581bc55a0ab1a44006421bbe02698d5b8 and run on the real config.json plus the real 2194 tensor names:

compressed-tensors refusal: []
IsMixedPrecision: 1
modelopt refusal: []

layers::modelopt::Refusal treats QuantAlgo::kNvfp4 and QuantAlgo::kW4A16Nvfp4 identically — one shared case accepting any module that ships an NVFP4 spelling — so the declaration passes its own cross-check. Routing is then done by the tensor-name probe, and VT_MODELOPT_W4A4 defaults to 0 because consuming input_scale produced incoherent text on nvidia/Qwen3.6-27B-NVFP4 (docs/ENVIRONMENT.md). The result is that we execute the weight-only W4A16 arm against a checkpoint whose producer declared static fp4 activations, and no line of output says so.

This is the live half of what the tree records as #1597 in .agents/completed/issue-index.md (that number does not resolve on the forge from this session, so treat its state as REMOTE_UNVERIFIED rather than closed), on a new artifact and with a consequence attached: it is now an unmatched axis in a published comparison rather than a latent one. It is also not a "wrong bytes" defect — the weight bytes swept per decode step are identical either way, so the roofline arithmetic in .agents/specs/bench-qwen38-27b-nvfp4-matched.md §2 is unaffected. What differs is the activation path and the numerics, and a token gate cannot see either, because W4A16 of a W4A4 checkpoint is numerically plausible.

What is wanted, in order of value.

  1. Say it. When the declaration is NVFP4 and the executed arm is W4A16, print one line naming both. Visible debt beats silent divergence, and it costs nothing.
  2. Decide whether VT_MODELOPT_W4A4 should be reachable on this checkpoint at all, given it is recorded as producing incoherent text on the 27B gate model. If our W4A4 path is wrong, that is a correctness gap on a shipped format, not a lever that happens to be off.
  3. The larger change #1597 already scoped (see the caveat above on its state): route by the declaration where one exists, keep the name probe where nothing is declared, and pay the same-binary A/B on nvidia/Qwen3.6-27B-NVFP4 that it owes, because the point of the change is that some module's arm moves.

The checkpoint is staged and pinned; its weights rows are in docs/USAGE.md with this change. Owed under ## Owed in .agents/specs/bench-qwen38-27b-nvfp4-matched.md.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the modelopt refusal and tensor-name routing code that selects the W4A16 or W4A4 arm, using the pinned config.json values and tests/vllm/models/qwen38_27b_modelopt_mtp_s1_manifest.inc as evidence. Read docs/ENVIRONMENT.md and the referenced .agents specs to understand the existing gate and owed comparison. Done should make the selected arm visible and establish how declaration-based routing and the same-binary A/B should be evaluated.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.