microsoft / microsoft/onnxruntime

I use the quantize_static to quantize float32 pytorch model to int8 model. However, it does not reduce the memory usage on CPU. Why?

Open
#15,106 5 comments 0 reactions 1 assignee Claimed by @jchen351 View on GitHub
quantization
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

I use the quantize_static to quantize float32 pytorch model to int8 model. However, it does not reduce the memory usage on CPU. Why?
The quantized model size is actually smaller, but the memory usage is almost the same (even slightly bigger) than the float32 model on CPU. #onnxruntime #quantization #quantize_static

### To reproduce

pytorch size : 1.5M float32 model

### Urgency

urgent for help

### Platform

Linux

### OS Version

ubuntu

### ONNX Runtime Installation

Built from Source

### ONNX Runtime Version or Commit ID

1.12

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

Default CPU

### Execution Provider Library Version

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.