microsoft / microsoft/onnxruntime

Static quantized resnet is slower than the raw one

Open
#5,319 13 comments 0 reactions 1 assignee View on GitHub

@yufenglee is already working on this.

Since Sep 29, 2020.

quantization
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

**Describe the bug**
Static quantized resnet is slower than the raw one.

Compared the static quantized Resnet model and the raw one from the E2E example code.
here is some benchmark from my computer

resnet50_v1.onnx / 3.03125
resnet50_v1.quant.onnx / 6.484375

here are the code piece
```

def perf_test(onnx_path, num=10):
sess = onnxruntime.InferenceSession(onnx_path)
name = sess.get_inputs()[0].name
data = np.random.random((1, 224, 224, 3)).astype(np.float32)
ort_in = {name: data}
# warm up
sess.run(None, ort_in)

latency = 0
for i in range(num):
start_32 = time.process_time()
ort_out = sess.run(None, ort_in)
latency = latency + time.process_time() - start_32
return latency

def latency_benchmark(num=10):
models = [
"resnet50_v1.onnx",
"resnet50_v1.quant.onnx",
]
latency_dict = {model: perf_test(model, num) for model in models}

for m, l in latency_dict.items():
print(f"{m} / {l}")

```

**Urgency**
Quantization performance

**System information**
Windows server
onnx==1.7.0
onnxruntime==1.4.0
Python version: 3.6

**To Reproduce**
See bug description

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.