microsoft / microsoft/onnxruntime

Quantization of T5-XL Model leads to "exceeds maximum protobuf"

Open
#7,974 5 comments 0 reactions 1 assignee View on GitHub

@yufenglee is already working on this.

Since Jun 7, 2021.

Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 8h
Merged PRs (30d)
179

Description

Hello,

I would like to quantize the T5-XL model (>2GB model size). It is basically the same problem as mentioned in #7017
There is a nightly solution proposed but I would really appreciate it if this would be supported natively.
Also mentioned in the above issue, the shape inference was removed in https://github.com/microsoft/onnxruntime/pull/5210/files, but it seems that this merge was reverted.

Describe the solution you'd like
In this post https://github.com/onnx/onnx/blob/master/docs/PythonAPIOverview.md#shape-inference-a-large-onnx-model-2gb is a currently supported variant of shape inference. So the current implementation of the shape inferences could be to use a try-exception-statement such that if the onnx.shape_inference.infer_shapes(model) fails, the onnx.shape_inference.infer_shapes_path('path/to/the/model.onnx') could be used. Although this is considered to not be good practice.
Further I am not sure if the shape inference is the only problem for large models.

Thank you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.