microsoft / microsoft/onnxruntime-genai
Builder '-m' does not support quantized models
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
**Describe the bug**
Passing HF model name through '-m' does not work when the model is a quantized model.
**To Reproduce**
Take https://huggingface.co/TheBloke/WizardLM-30B-GPTQ for example.
Running the command `python -m onnxruntime_genai.models.builder -m TheBloke/WizardLM-30B-GPTQ -p int4 -e cpu -o wizardlm_30b_gptq` emits error
```
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/builder.py", line 2645, in
create_model(args.model_name, args.input, args.output, args.precision, args.execution_provider, args.cache_dir, **extra_options)
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/builder.py", line 2538, in create_model
onnx_model.make_model(input_path)
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/builder.py", line 1623, in make_model
model = QuantModel.from_pretrained(self.quant_type, input_path, self.quant_attrs["bits"], self.quant_attrs["group_size"], self.quant_attrs["use_g_idx"], q_size, kv_size, self.intermediate_size)
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/quantized_model.py", line 654, in from_pretrained
model = GPTQModel(quant_type, input_path, bits, group_size, use_g_idx, q_size, kv_size, intermediate_size)
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/quantized_model.py", line 590, in __init__
super().__init__(quant_type, input_path, bits, group_size, q_size, kv_size, intermediate_size)
File "/opt/conda/envs/torch/lib/python3.10/site-packages/onnxruntime_genai/models/quantized_model.py", line 94, in __init__
for weight_file in os.listdir(input_path):
FileNotFoundError: [Errno 2] No such file or directory: ''
```
**Expected behavior**
Command to succeed with generated ONNX model.
> onnxruntime-genai ffc0fd4c5ef4cec037c04175eea57589023c1eed
cc @kunal-vaishnavi
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with models/builder.py and models/quantized_model.py at the stack-trace locations, then run the reported builder command with the WizardLM-30B-GPTQ model. Trace how the -m model name is converted into the input path for quantized models; done means the command succeeds and produces the expected ONNX model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100