microsoft / microsoft/onnxruntime
I use the quantize_static to quantize float32 pytorch model to int8 model. However, it does not reduce the memory usage on CPU. Why?
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
I use the quantize_static to quantize float32 pytorch model to int8 model. However, it does not reduce the memory usage on CPU. Why?
The quantized model size is actually smaller, but the memory usage is almost the same (even slightly bigger) than the float32 model on CPU. #onnxruntime #quantization #quantize_static
### To reproduce
pytorch size : 1.5M float32 model
### Urgency
urgent for help
### Platform
Linux
### OS Version
ubuntu
### ONNX Runtime Installation
Built from Source
### ONNX Runtime Version or Commit ID
1.12
### ONNX Runtime API
Python
### Architecture
X64
### Execution Provider
Default CPU
### Execution Provider Library Version
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.