NVIDIA / NVIDIA/Model-Optimizer

NVFP4 support for Qwen3_5MoeExperts (fused MoE): quantizer registration + export serializer

Open
#2,011 8 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.8k
Forks
604
Avg merge
2d 8h
Merged PRs (30d)
142

Description

Summary

mtq.quantize silently skips the fused experts of transformers Qwen3_5MoeExperts (Qwen3.5/3.6-VL-MoE) — they're 3-D nn.Parameters (gate_up_proj [E,2I,H], down_proj [E,H,I]) with a per-expert F.linear loop, and there's no QuantModuleRegistry entry. Result: the ~50GB expert bulk stays bf16, and export_hf_checkpoint has no serializer for it (layer_utils.get_experts_list only handles per-expert nn.Linear, Mixtral/DBRX-style).

Proposed

  1. Register a _QuantQwen3_5MoeExperts(QuantModule) (analogous to _QuantLlama4TextExperts but with F.linear/(out,in) layout — no transpose). Reference impl that works today as an external registration is included in the HF repo below.
  2. Add a fused-3-D export path so export_hf_checkpoint emits w13/w2 NVFP4 (packed uint8 + fp8 block scale + fp32 per-shard global + input scale).

Workaround shipped

Custom registration + manual NVFP4 packing via NVFP4QTensor (global amax/(6·448), block amax/(6·global)), writing the vLLM-ready checkpoint directly (also avoids export_hf_checkpoint OOM on unified-memory boxes).

Artifacts

Full quantize.py + resulting model (67GB→22GB, vision intact): https://huggingface.co/frischeDaten/Qwen3.6-VL-35B-A3B-NVFP4-DGX-Spark-VisionSafe

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.