microsoft / microsoft/onnxruntime-inference-examples
Issue: Error Loading 8 bit quntized ONNX Model on inference- Protobuf Parsing Failed
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.7k
- Forks
- 414
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 14
Description
Description
I am facing an error when attempting to load a quantized ONNX model using the ORTModelForCausalLM class from the optimum.onnxruntime library. The error message states: "Failed to load model because protobuf parsing failed."
Context
I am using dynamic quantization for 8-bit quantization on the model before loading it for inference with ONNX.
from optimum.onnxruntime import ORTModelForCausalLM
from transformers import pipeline, LlamaTokenizer
import torch
onnx_path = "./llmonnx/"
opt_model = ORTModelForCausalLM.from_pretrained(onnx_path, file_name="model.onnx").to('cuda')
tokenizer = LlamaTokenizer.from_pretrained(onnx_path)
opt_optimum_generator = pipeline("text-generation", model=opt_model, tokenizer=tokenizer, device='cuda')
prompt = "give me translation for this?"
generated_text = opt_optimum_generator(prompt, max_length=512, num_return_sequences=1, truncation=True)
print(generated_text[0]['generated_text']) please assist me if there is any resolution for this Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided ORTModelForCausalLM.from_pretrained example and the reported protobuf parsing error. Reproduce the failure with the dynamically quantized model and inspect the model-loading path; done means identifying a confirmed cause and documenting or validating a resolution. No repository file or test is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100