My Quantized model not running faster than Unquantized model.

Hello @edgchen1 @wejoncy I tried to quantize the mars-model used in deepsort tracking. Using the example in image_classification/cpu I am able to quantize my mars model. Size of the model has reduced after quantization. But inference speed of the quantized model has not increased. It is very much similar to my unquantized model. What could be the problem here? I will mention the steps I did to quantize.

Firstly. mars model used in deepsort repo is a tensorflow .pb model. I took that model and then converted it into onnx format using tf2onnx utility. Now on this onnx model, I have applied static quantization as described in the example under quantization/image_classification/cpu. I successfully got the quantized onnx model which is smaller in size. But the issue is with inference speed which has not increased. Any help is appreciated.

Start by reviewing the quantization example under quantization/image_classification/cpu and the conversion step using tf2onnx. Reproduce inference with the original and quantized mars-model ONNX files, then compare the measurement setup and document what explains the missing speed improvement.