microsoft / microsoft/onnxruntime

[Performance] QuantizeLinear/DequantizeLinear node does not support per-channel on GPU?

Open
#15,260 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

ep:CUDA feature request quantization
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

Describe the feature request

QuantizeLinear/DequantizeLinear CUDA kernels do not support per-channel

Describe scenario use case

In order to fit a larger model without loss of accuracy when the GPU memory is limited, I want to perform int8 quantization only on the weights.
 However, when I infer the model, I found that the DequantizeLinear node takes a lot of time (as shown in the figure below) because the op is not running by the CUDAExecutionProvider.
Considering that the QuantizeLinear/DequantizeLinear CUDA kernels do not support per-channel version implementation, I intend to contribute a version of Q/DQ that supports per-channel functions to the community. Not sure if it is suitable?

Q/DQ implementation onnxruntime/core/providers/cuda/tensor/quantize_linear.cc
image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with onnxruntime/core/providers/cuda/tensor/quantize_linear.cc and inspect how the CUDA QuantizeLinear and DequantizeLinear kernels handle scales. Determine the per-channel behavior required for both operators and how CUDAExecutionProvider dispatches them. Done means per-channel Q/DQ operations run through the CUDA kernels rather than falling back, with appropriate coverage verified by the project's existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.