NVIDIA / NVIDIA/Model-Optimizer
NVFP4 support for Qwen3_5MoeExperts (fused MoE): quantizer registration + export serializer
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 604
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 142
Description
Summary
mtq.quantize silently skips the fused experts of transformers Qwen3_5MoeExperts (Qwen3.5/3.6-VL-MoE) — they're 3-D nn.Parameters (gate_up_proj [E,2I,H], down_proj [E,H,I]) with a per-expert F.linear loop, and there's no QuantModuleRegistry entry. Result: the ~50GB expert bulk stays bf16, and export_hf_checkpoint has no serializer for it (layer_utils.get_experts_list only handles per-expert nn.Linear, Mixtral/DBRX-style).
Proposed
- Register a
_QuantQwen3_5MoeExperts(QuantModule)(analogous to_QuantLlama4TextExpertsbut withF.linear/(out,in)layout — no transpose). Reference impl that works today as an external registration is included in the HF repo below. - Add a fused-3-D export path so
export_hf_checkpointemitsw13/w2NVFP4 (packed uint8 + fp8 block scale + fp32 per-shard global + input scale).
Workaround shipped
Custom registration + manual NVFP4 packing via NVFP4QTensor (global amax/(6·448), block amax/(6·global)), writing the vLLM-ready checkpoint directly (also avoids export_hf_checkpoint OOM on unified-memory boxes).
Artifacts
Full quantize.py + resulting model (67GB→22GB, vision intact): https://huggingface.co/frischeDaten/Qwen3.6-VL-35B-A3B-NVFP4-DGX-Spark-VisionSafe
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.