QUANT-QWEN38-27B-NVFP4-ARM: qwen36 compressed-tensors mixed-precision FP8 group_0 is refused, blocking an apples-to-apples vllm.cpp vs vLLM comparison
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Blocker
/data/models/Qwen/Qwen3.6-27B-NVFP4 (the checkpoint nsk/vLLM serves) and its
-vllmcpp sibling declare quantization_config = compressed-tensors,
format: "mixed-precision", with two groups:
group_0: FP8float-quantized, per-output-channelweight_scaleand
DYNAMIC per-token activation quantization (input_activations.dynamic: true, strategy: token).group_1: NVFP4nvfp4-pack-quantized.
Current main loads group_1 but refuses group_0 in
qwen3_5_dense_weights.cpp via compressed_tensors::RefusalForHfConfigRaw
(because LoadFp8Raw cannot represent a per-channel scale + dynamic activation).
So vllm-server exits fatal before reading a weight byte.
Why it matters
The vllm.cpp vs vLLM (nsk) performance comparison is currently cross-model
(vllm.cpp runs the AEON MTP checkpoint, vLLM runs qwen36). To make it
apples-to-apples, vllm.cpp must load the SAME qwen36 checkpoint vLLM serves.
Goal
Implement the compressed-tensors FP8 float-quantized per-channel + dynamic
activation arm so qwen36 loads, then re-run the identical evalscope random
matrix on both engines over the same checkpoint.
Owning row
QUANT-QWEN38-27B-NVFP4-ARM (the loader's compressed-tensors mixed-precision
resolution; spec .agents/specs/qwen38-27b-quant-arms.md)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in qwen3_5_dense_weights.cpp at compressed_tensors::RefusalForHfConfigRaw and inspect how LoadFp8Raw handles the checkpoint configuration. Read .agents/specs/qwen38-27b-quant-arms.md and compare the group_0 FP8 settings with the existing mixed-precision path. Done means the same qwen36 checkpoint loads in vllm-server and the identical evalscope random matrix can be rerun against vLLM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai-infra-agents
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100