NVIDIA / NVIDIA/TensorRT-LLM

[Feature]: Expose SmoothQuant alpha for int8_sq quantization in quantize.py

Open
#16,151 0 comments 0 reactions 1 assignee View on GitHub

@laikhtewari is already working on this.

Since Jul 8, 2026.

feature request Model optimization
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

🚀 The feature, motivation and pitch

ModelOpt's SmoothQuant calibration defaults to alpha=1.0 and TensorRT-LLM's
examples/quantization/quantize.py exposes no way to change it — the only override in the
tree is the hardcoded Gemma special case in quantize_by_modelopt.py
({"method": "smoothquant", "alpha": 0.5}). The SmoothQuant paper recommends 0.5–0.9
depending on the model.

In our sweeps on Qwen3-0.6B/1.7B (11 alphas × 2 calib sets, real TRT engines, GSM8K-1319),
alpha=1.0 was never optimal and cost up to 20 GSM8K points vs the best alpha in the same
setting (worst case: Qwen3-0.6B, 15.39% at alpha=1.0 vs 35.41% at alpha=0.85, against a
36.32% bf16 baseline).

Proposal: add --smoothquant_alpha (float, default None = current behavior) to quantize.py
and thread it to quantize_and_export, where it sets the same algorithm dict the Gemma
special case already uses. Backward compatible; ModelOpt already pydantic-validates the
field (bounds [0, 1]). Happy to submit the PR (change is ready).

Alternatives

ModelOpt's hf_ptq.py has a --recipe mechanism, so an alternative is to contribute a recipe
YAML on the Model-Optimizer side instead. But TRT-LLM's quantize.py has no recipe mechanism,
so exposing the flag directly is the minimal, self-contained fix for this CLI. The two
approaches don't conflict.

Additional context

Full sweep (11 alphas × 2 calib sets × 2 Qwen3 sizes):

Model Calib alpha=1.0 (current) best alpha GSM8K cost of default
Qwen3-1.7B wikitext 39.80% 57.09% @ α=0.75 −17.3 pts
Qwen3-1.7B gsm8k-train 61.87% 63.76% @ α=0.85 −1.9 pts
Qwen3-0.6B wikitext 21.76% 31.54% @ α=0.75 −9.8 pts
Qwen3-0.6B gsm8k-train 15.39% 35.41% @ α=0.85 −20.0 pts

The PR is ready: +121/−1, includes a GPU-free unit test
(tests/unittest/trt/quantization/test_smoothquant_alpha.py). No behavior change when the
flag is unset.

Before submitting a new issue...
  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.