microsoft / microsoft/onnxruntime-genai
`onnxruntime-genai` generation speed very slow on int4
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
I have built a small example using the python binding here https://github.com/tarekziade/onnxruntime-test/blob/main/run.py
to measure the inference speed on my Apple M1 and on a windows 11 box, using Qwen 2.5 0.5B instruct
to prepare the model I used the cpu provider and int4/fp16/fp32 precisions:
```
python3 -m onnxruntime_genai.models.builder -m "Qwen/Qwen2.5-0.5B-Instruct" -o qwen -p int4 -e cpu
python3 -m onnxruntime_genai.models.builder -m "Qwen/Qwen2.5-0.5B-Instruct" -o qwen -p fp32 -e cpu
python3 -m onnxruntime_genai.models.builder -m "Qwen/Qwen2.5-0.5B-Instruct" -o qwen -p fp16 -e cpu
```
And compared the execution times with llama-cli using a GGUF of the same model using q4_0


One apple, the int4 precision is extremely slow on and fp16 failed on both platforms with
```bash
onnxruntime_genai.onnxruntime_genai.OrtException:
Non-zero status code returned while running Cast node.
Name:'InsertedPrecisionFreeCast_/model/layers.1/attn/v_proj/repeat_kv/Reshape_4/output_0' Status
Message: /Users/runner/work/1/s/onnxruntime/core/framework/op_kernel.cc:83 virtual OrtValue *onnxruntime::OpKernelContext::OutputMLValue(int, const onnxruntime::TensorShape &) status.IsOK() was false.
Shape mismatch attempting to re-use buffer. {1,1,896} != {1,248,896}.
Validate usage of dim_value (values should be > 0) and dim_param (all values with the same string should equate to the same size) in shapes in the model.
```
I was wondering if I did something wrong? I was also wondering if int8 precision is an option. looks like onnxruntime_genai.models.builder can use some int8 quantizations using the int4 mode but I am not entirely clear about this
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked run.py and reproduce the builder commands for Qwen2.5-0.5B-Instruct on CPU, comparing int4, fp16, and fp32 with the reported llama-cli results. Investigate the int4 performance and the fp16 Cast shape-mismatch error; done should identify whether the behavior is expected, a reproducible defect, or a supported-precision/documentation gap.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100