NVIDIA / NVIDIA/Model-Optimizer

Why introduce the design of an `output_quantizer` within quantization operators?

Open
#1,533 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
3.8k
Forks
604
Avg merge
2d 8h
Merged PRs (30d)
142

Description

I'm puzzled by a specific question: why does modelopt introduce the design of an output_quantizer within QuantOP(QuantInputBase, QuantLinearConvBase), then disable output_quantizer?(code from here)

I think there might be two places that could be the cause:

  1. For kv cache quantization:
  • run_auto_quantize.py set "*output_quantizer" to enable.(code from here)
  • quantization/algorithms.py set "*output_quantizer" to false to diable kv cache quantization. (code from here)

However it now appears that configuring KV cache quantization is achieved via "*[kv]_bmm_quantizer".

  1. For LayerNorm output, _FP8_MHA_OVERRIDE set "*output_quantizer" to enable(code from here). I am unclear on what "fuse the shared Q/DQ across all downstream Q/K/V/FC consumers" implies. Does this mean that the Q, K, V, and FC inputs are all expected to be FP8-quantized?

Are there other reasons related to the definition of output_quantizer that I am unaware of? Looking forward to your reply, Thanks in advance!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with modelopt/torch/quantization/nn/modules/quant_module.py and compare the output_quantizer handling with the referenced examples/llm_autodeploy/run_auto_quantize.py, modelopt/torch/quantization/algorithms.py, and examples/torch_onnx/torch_quant_to_onnx.py. Trace the KV-cache and FP8 MHA configurations, then document the rationale and clarify what behavior is expected when output_quantizer is enabled or disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.