microsoft / microsoft/onnxruntime

MaxPool not getting quantized when preceded by Relu

Open
#9,428 3 comments 0 reactions 1 assignee View on GitHub

@zhanghuanrong is already working on this.

Since Oct 18, 2021.

quantization
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 8h
Merged PRs (30d)
179

Description

Describe the bug
MaxPool nodes are not getting quantized if a preceding Relu is not getting quantized.

Urgency

Development of a backend is blocked by this, so it would be great if someone could provide some insights as soon as possible.

System information

  • ONNX Runtime installed from (source or binary): source
  • ONNX Runtime version: 1.9.1
  • Python version: 3.8

To Reproduce

  • Run script below and inspect resulting quantized model

Expected behavior

Would expect a QuantizeLinear node before the Maxpool node in the quantized model.

import torch
import numpy as np
import onnx
import onnxruntime
from onnxruntime import quantization

IMAGE_SHAPE  = (1, 1, 16, 16)
KERNEL_SHAPE = (1, 1,  2,  2)

class ToyModel(torch.nn.Module):

    def __init__(self):
        super().__init__()
        self.conv = torch.nn.Conv2d(out_channels = KERNEL_SHAPE[0],
                                    in_channels  = KERNEL_SHAPE[1],
                                    kernel_size  = KERNEL_SHAPE[2:],
                                    bias         = False)
        weight = torch.tensor(data  = 127*np.ones(KERNEL_SHAPE).astype("float32"),
                              dtype = torch.float32)
        self.conv.weight = torch.nn.Parameter(weight, requires_grad = False)
        self.max_pool = torch.nn.MaxPool2d(kernel_size = (2,2))

    def forward(self, input):
        return self.max_pool(self.conv(input).relu())

class ToyDataProvider(onnxruntime.quantization.CalibrationDataReader):

    def __init__(self, input_name):
        self.data = ({ input_name: prefac*np.ones(IMAGE_SHAPE).astype("float32") } for prefac in [-127,+127])

    def get_next(self):
        try: return next(self.data)
        except StopIteration: return None

def CreateToyModels(unquantized_path, quantized_path):

    # Save toy model to onnx file
    model = ToyModel()
    torch.onnx.export(model, (torch.empty(IMAGE_SHAPE, dtype=torch.float32)), unquantized_path)
    session = onnxruntime.InferenceSession(unquantized_path)
    input_name = session.get_inputs()[0].name

    # Quantize model with Relu using dummy data
    onnxruntime.quantization.quantize_static(model_input = unquantized_path,
                                             model_output = quantized_path,
                                             calibration_data_reader = ToyDataProvider(input_name),
                                             activation_type = onnxruntime.quantization.QuantType.QUInt8,
                                             weight_type = onnxruntime.quantization.QuantType.QUInt8,
                                             op_types_to_quantize = ["Conv", "MaxPool"],
                                             extra_options = { "WeightSymmetric" : False,
                                                               "ActivationSymmetric" : False })
                                             
if __name__ == "__main__":

    # Create quantized and unquantized models
    CreateToyModels(unquantized_path = "model.onnx", quantized_path = "quantized-model.onnx")
    

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.