microsoft / microsoft/onnxruntime
Quantization of T5-XL Model leads to "exceeds maximum protobuf"
@yufenglee is already working on this.
Since Jun 7, 2021.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 179
Description
Hello,
I would like to quantize the T5-XL model (>2GB model size). It is basically the same problem as mentioned in #7017
There is a nightly solution proposed but I would really appreciate it if this would be supported natively.
Also mentioned in the above issue, the shape inference was removed in https://github.com/microsoft/onnxruntime/pull/5210/files, but it seems that this merge was reverted.
Describe the solution you'd like
In this post https://github.com/onnx/onnx/blob/master/docs/PythonAPIOverview.md#shape-inference-a-large-onnx-model-2gb is a currently supported variant of shape inference. So the current implementation of the shape inferences could be to use a try-exception-statement such that if the onnx.shape_inference.infer_shapes(model) fails, the onnx.shape_inference.infer_shapes_path('path/to/the/model.onnx') could be used. Although this is considered to not be good practice.
Further I am not sure if the shape inference is the only problem for large models.
Thank you.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.