microsoft / microsoft/onnxruntime-inference-examples

Issue: Error Loading 8 bit quntized ONNX Model on inference- Protobuf Parsing Failed

Open
#453 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.7k
Forks
414
Avg merge
1d 6h
Merged PRs (30d)
14

Description

Description

I am facing an error when attempting to load a quantized ONNX model using the ORTModelForCausalLM class from the optimum.onnxruntime library. The error message states: "Failed to load model because protobuf parsing failed."
Context

I am using dynamic quantization for 8-bit quantization on the model before loading it for inference with ONNX.
protobuf_decoding from optimum.onnxruntime import ORTModelForCausalLM
from transformers import pipeline, LlamaTokenizer
import torch

onnx_path = "./llmonnx/"
opt_model = ORTModelForCausalLM.from_pretrained(onnx_path, file_name="model.onnx").to('cuda')
tokenizer = LlamaTokenizer.from_pretrained(onnx_path)
opt_optimum_generator = pipeline("text-generation", model=opt_model, tokenizer=tokenizer, device='cuda')

prompt = "give me translation for this?"
generated_text = opt_optimum_generator(prompt, max_length=512, num_return_sequences=1, truncation=True)
print(generated_text[0]['generated_text']) please assist me if there is any resolution for this Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided ORTModelForCausalLM.from_pretrained example and the reported protobuf parsing error. Reproduce the failure with the dynamically quantized model and inspect the model-loading path; done means identifying a confirmed cause and documenting or validating a resolution. No repository file or test is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.