mudler / mudler/vllm.cpp

QUANT-QWEN38-27B-NVFP4-ARM: qwen36 compressed-tensors mixed-precision FP8 group_0 is refused, blocking an apples-to-apples vllm.cpp vs vLLM comparison

Open
#1,959 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Blocker

/data/models/Qwen/Qwen3.6-27B-NVFP4 (the checkpoint nsk/vLLM serves) and its
-vllmcpp sibling declare quantization_config = compressed-tensors,
format: "mixed-precision", with two groups:

  • group_0: FP8 float-quantized, per-output-channel weight_scale and
    DYNAMIC per-token activation quantization (input_activations.dynamic: true, strategy: token).
  • group_1: NVFP4 nvfp4-pack-quantized.

Current main loads group_1 but refuses group_0 in
qwen3_5_dense_weights.cpp via compressed_tensors::RefusalForHfConfigRaw
(because LoadFp8Raw cannot represent a per-channel scale + dynamic activation).
So vllm-server exits fatal before reading a weight byte.

Why it matters

The vllm.cpp vs vLLM (nsk) performance comparison is currently cross-model
(vllm.cpp runs the AEON MTP checkpoint, vLLM runs qwen36). To make it
apples-to-apples, vllm.cpp must load the SAME qwen36 checkpoint vLLM serves.

Goal

Implement the compressed-tensors FP8 float-quantized per-channel + dynamic
activation arm so qwen36 loads, then re-run the identical evalscope random
matrix on both engines over the same checkpoint.

Owning row

QUANT-QWEN38-27B-NVFP4-ARM (the loader's compressed-tensors mixed-precision
resolution; spec .agents/specs/qwen38-27b-quant-arms.md)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in qwen3_5_dense_weights.cpp at compressed_tensors::RefusalForHfConfigRaw and inspect how LoadFp8Raw handles the checkpoint configuration. Read .agents/specs/qwen38-27b-quant-arms.md and compare the group_0 FP8 settings with the existing mixed-precision path. Done means the same qwen36 checkpoint loads in vllm-server and the identical evalscope random matrix can be rerun against vLLM.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai-infra-agents
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.