microsoft / microsoft/onnxruntime-genai

Builder '-m' does not support quantized models

Open
#771 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**
Passing HF model name through '-m' does not work when the model is a quantized model.

**To Reproduce**
Take https://huggingface.co/TheBloke/WizardLM-30B-GPTQ for example.

Running the command `python -m onnxruntime_genai.models.builder -m TheBloke/WizardLM-30B-GPTQ -p int4 -e cpu -o wizardlm_30b_gptq` emits error

```
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/builder.py", line 2645, in
create_model(args.model_name, args.input, args.output, args.precision, args.execution_provider, args.cache_dir, **extra_options)
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/builder.py", line 2538, in create_model
onnx_model.make_model(input_path)
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/builder.py", line 1623, in make_model
model = QuantModel.from_pretrained(self.quant_type, input_path, self.quant_attrs["bits"], self.quant_attrs["group_size"], self.quant_attrs["use_g_idx"], q_size, kv_size, self.intermediate_size)
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/quantized_model.py", line 654, in from_pretrained
model = GPTQModel(quant_type, input_path, bits, group_size, use_g_idx, q_size, kv_size, intermediate_size)
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/quantized_model.py", line 590, in __init__
super().__init__(quant_type, input_path, bits, group_size, q_size, kv_size, intermediate_size)
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/quantized_model.py", line 94, in __init__
for weight_file in os.listdir(input_path):
FileNotFoundError: [Errno 2] No such file or directory: ''
```

**Expected behavior**
Command to succeed with generated ONNX model.

> onnxruntime-genai ffc0fd4c5ef4cec037c04175eea57589023c1eed

cc @kunal-vaishnavi

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with models/builder.py and models/quantized_model.py at the stack-trace locations, then run the reported builder command with the WizardLM-30B-GPTQ model. Trace how the -m model name is converted into the input path for quantized models; done means the command succeeds and produces the expected ONNX model.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.