microsoft / microsoft/onnxruntime-genai

`onnxruntime-genai` generation speed very slow on int4

Open
#1,098 10 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

I have built a small example using the python binding here https://github.com/tarekziade/onnxruntime-test/blob/main/run.py
to measure the inference speed on my Apple M1 and on a windows 11 box, using Qwen 2.5 0.5B instruct

to prepare the model I used the cpu provider and int4/fp16/fp32 precisions:

```
python3 -m onnxruntime_genai.models.builder -m "Qwen/Qwen2.5-0.5B-Instruct" -o qwen -p int4 -e cpu
python3 -m onnxruntime_genai.models.builder -m "Qwen/Qwen2.5-0.5B-Instruct" -o qwen -p fp32 -e cpu
python3 -m onnxruntime_genai.models.builder -m "Qwen/Qwen2.5-0.5B-Instruct" -o qwen -p fp16 -e cpu
```

And compared the execution times with llama-cli using a GGUF of the same model using q4_0

![Apple M1](https://raw.githubusercontent.com/tarekziade/onnxruntime-test/refs/heads/main/macos.png)
![Windows 11](https://raw.githubusercontent.com/tarekziade/onnxruntime-test/refs/heads/main/win11.png)

One apple, the int4 precision is extremely slow on and fp16 failed on both platforms with

```bash
onnxruntime_genai.onnxruntime_genai.OrtException:
Non-zero status code returned while running Cast node.
Name:'InsertedPrecisionFreeCast_/model/layers.1/attn/v_proj/repeat_kv/Reshape_4/output_0' Status
Message: /Users/runner/work/1/s/onnxruntime/core/framework/op_kernel.cc:83 virtual OrtValue *onnxruntime::OpKernelContext::OutputMLValue(int, const onnxruntime::TensorShape &) status.IsOK() was false.
Shape mismatch attempting to re-use buffer. {1,1,896} != {1,248,896}.
Validate usage of dim_value (values should be > 0) and dim_param (all values with the same string should equate to the same size) in shapes in the model.
```

I was wondering if I did something wrong? I was also wondering if int8 precision is an option. looks like onnxruntime_genai.models.builder can use some int8 quantizations using the int4 mode but I am not entirely clear about this

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked run.py and reproduce the builder commands for Qwen2.5-0.5B-Instruct on CPU, comparing int4, fp16, and fp32 with the reported llama-cli results. Investigate the int4 performance and the fp16 Cast shape-mismatch error; done should identify whether the behavior is expected, a reproducible defect, or a supported-precision/documentation gap.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.