microsoft / microsoft/onnxruntime-genai
qwen2.5 output is not correct under dml ep
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
**Describe the bug**
model: Qwen2.5-7B-Instruct convert to onnx model using onnxruntime_genai.models.builder
when use cpu ep : the output is correct:

but the output of directml ep is abnormal:

**To Reproduce**
1. commoad covert model:
python -m onnxruntime_genai.models.builder -m "D:\models\Qwen2.5-7B-Instruct" -e dml -p int4 -o "D:\models\Qwen2.5-7B-Instruct-onnx-dml"
python -m onnxruntime_genai.models.builder -m "D:\models\Qwen2.5-7B-Instruct" -e cpu -p int4 -o "D:\models\Qwen2.5-7B-Instruct-onnx-cpu"
2. test python code:
```python
import onnxruntime_genai as og
from transformers import AutoTokenizer
model = og.Model("D:\models\Qwen2.5-7B-Instruct-onnx")
tokenizer = og.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()
# tokenizer = AutoTokenizer.from_pretrained("D:\models\Qwen2.5-7B-Instruct-onnx-dml")
# Set the max length to something sensible by default,
# since otherwise it will be set to the entire context length
search_options = {"max_length": 2048, 'do_sample': True, 'top_k': 40, 'temperature': 0.2, 'top_p': 0.8,
'repetition_penalty': 1.2}
chat_template = (
"\n<|im_start|>user\n{}<|im_end|>"
"\n<|im_start|>assistant\n"
)
prompt = f'{chat_template.format("tell a story about your favorite food!")}'
print(prompt)
input_tokens = tokenizer.encode(prompt)
params = og.GeneratorParams(model)
params.set_search_options(**search_options)
params.input_ids = input_tokens
generator = og.Generator(model, params)
print("Output: ", end='', flush=True)
import time
s = time.time()
n_tks = 0
try:
while not generator.is_done():
generator.compute_logits()
generator.generate_next_token()
new_token = generator.get_next_tokens()[0]
n_tks += 1
print(tokenizer_stream.decode(new_token), end='', flush=True)
e = time.time()
print(f"\n{n_tks / (e - s):.2f} tks/s")
except KeyboardInterrupt:
e = time.time()
print(f"\n{n_tks / (e - s):.2f} tks/s")
print(" --control+c pressed, aborting generation--")
print()
del generator
```
**env**
- intel Ultra 7 155H igpu , driver version : 32.0.101.5972
- OS: windows11 , python3.11 , onnxruntime-dml 1.19.2 , onnxruntime-genai-directml 0.5.0.rc1 ,
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the two `onnxruntime_genai.models.builder` conversions and the supplied Python generation script, comparing CPU and DirectML results. Investigate the DirectML path and the reported environment versions; done means Qwen2.5 generation produces correct output under the DirectML execution provider.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100