microsoft / microsoft/onnxruntime

[Feature Request] ONNX lossy+lossless compression.

Open
#15,456 4 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

**Here is the proposal:**

If we have an ONNX file of float16s. This might be a file of 10GB.
Let's say we "quantize" these into int8's. So now the file is of size 5GB.
But we don't run it as int8.
When we load the InferenceSession, these weights get turned back into float16s.

So basically this is just a float16 ONNX file that is compressed on the disk. (Much like a GIF or JPEG is a compressed image file)

It would be cool to have some built in lossy and lossless compression functions specifically designed for neural networks. That would create smaller ONNX files.

If we just ZIP it with standard compressors, then this is not optimal and very bad for resources.

If this is not a good fit for the OnnxRuntime. Perhaps you might like to suggest some good compression algorithms that would work well with neural networks?

### Describe scenario use case

When sharing an application people would need to download smaller files and save hard disk space. The files would be decompressed sequentially at runtime and become full size on the GPU before inference. (Using an efficient loader such as [this](https://github.com/pauldog/FastOnnxLoader)).

float16 or float32 often work faster on GPU's than int8. So it would be nice to store the values on disk as int8 but run them on the GPU as float16.

Personally I am having problem in that my 256GB SSD is getting full of very large >5GB ONNX files. So some sort of compression would be appreciated.

Basically I am proposing a "Compressed-ONNX" format (both lossy and lossless) together with the functions to decompress it at runtime. I will probably end up implementing this myself but it is just a suggestion for you.

### Ideal Solutoin
The ideal solution would be to have an onnx compressed in int8 format. And then you could choose whether to run this on the GPU as int8, float16 or float32. I think this is possible since you could read in the "DeQuantize layers" use these to convert the int8 back to float before inference. In other words there should be a DeQuantize(model) function.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. The stated goal is a compressed ONNX format with runtime decompression and selectable int8, float16, or float32 execution, but the issue does not define an implementation scope or completion test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, machine-learning
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.