NVIDIA / NVIDIA/Model-Optimizer

Add VLM components to _default_disabled_quantizer_cfg?

Open
#1,396 3 comments 0 reactions 1 assignee View on GitHub

@Edwardf0t1 is already working on this.

Since May 7, 2026.

feature request
Dominant language
Python
Stars
3.8k
Forks
604
Avg merge
2d 8h
Merged PRs (30d)
142

Description

Hello, I'd like to propose a small change to how multimodal models are handled during quantization. Please correct me if my understanding is incorrect.

Currently, when quantizing a multimodal model with any recipe, ModelOpt attaches observers to the vision tower and multimodal projector alongside the language model. For models like moonshotai/Kimi-K2.6, this breaks at export with ValueError: tensor column shape must be divisible by the given group_size 32 but got 4304 (raised by compressed_tensors/quantization/lifecycle/forward_helpers.py:138 from inside modelopt.export_hf_checkpoint)". The failure happens after calibration completes, so the issue only happens after the run.

Every published VLM NVFP4 checkpoint I could find like nvidia/Kimi-K2.5-NVFP4, wafer-ai/Kimi-K2.6-NVFP4, RedHatAI/Kimi-K2.6-NVFP4 keeps vision and projector layers in BF16. I think intuitively this makes sense because vision encoders are tiny on large VLMs (typically <1% of params), so quantizing them yields negligible memory savings and quantization noise at the visual feature stage compounds through the LLM layers.

I think we can append ~4 patterns to _default_disabled_quantizer_cfg, this is the same universal-disable list that already covers lm_head, MoE routers, BatchNorm, and Mamba conv1d. This makes sense to me intuitively because looking at the code, the convention there seems to be "always wrong to quantize regardless of recipe," and I think VLM vision/projector components appear to meet that bar?

I had to fix these issues to get a working Kimi2.6 quant that serves inference via SGLang. Happy to provide details if needed and make a PR!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.