microsoft / microsoft/onnxruntime-genai

qwen2.5 output is not correct under dml ep

Open
#1,019 1 comment 0 reactions 0 assignees View on GitHub
ep:DML model quality
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**

model: Qwen2.5-7B-Instruct convert to onnx model using onnxruntime_genai.models.builder

when use cpu ep : the output is correct:
![Image](https://github.com/user-attachments/assets/b2b818e6-5d7f-4d77-ba6f-43af1810a11b)

but the output of directml ep is abnormal:
![Image](https://github.com/user-attachments/assets/53111253-6245-464d-b144-b05a232b0016)

**To Reproduce**
1. commoad covert model:
python -m onnxruntime_genai.models.builder -m "D:\models\Qwen2.5-7B-Instruct" -e dml -p int4 -o "D:\models\Qwen2.5-7B-Instruct-onnx-dml"
python -m onnxruntime_genai.models.builder -m "D:\models\Qwen2.5-7B-Instruct" -e cpu -p int4 -o "D:\models\Qwen2.5-7B-Instruct-onnx-cpu"

2. test python code:

```python

import onnxruntime_genai as og
from transformers import AutoTokenizer

model = og.Model("D:\models\Qwen2.5-7B-Instruct-onnx")
tokenizer = og.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()
# tokenizer = AutoTokenizer.from_pretrained("D:\models\Qwen2.5-7B-Instruct-onnx-dml")

# Set the max length to something sensible by default,
# since otherwise it will be set to the entire context length
search_options = {"max_length": 2048, 'do_sample': True, 'top_k': 40, 'temperature': 0.2, 'top_p': 0.8,
'repetition_penalty': 1.2}

chat_template = (
"\n<|im_start|>user\n{}<|im_end|>"
"\n<|im_start|>assistant\n"
)

prompt = f'{chat_template.format("tell a story about your favorite food!")}'

print(prompt)
input_tokens = tokenizer.encode(prompt)

params = og.GeneratorParams(model)
params.set_search_options(**search_options)
params.input_ids = input_tokens
generator = og.Generator(model, params)

print("Output: ", end='', flush=True)

import time

s = time.time()
n_tks = 0

try:
while not generator.is_done():
generator.compute_logits()
generator.generate_next_token()

new_token = generator.get_next_tokens()[0]
n_tks += 1
print(tokenizer_stream.decode(new_token), end='', flush=True)
e = time.time()
print(f"\n{n_tks / (e - s):.2f} tks/s")
except KeyboardInterrupt:
e = time.time()
print(f"\n{n_tks / (e - s):.2f} tks/s")
print(" --control+c pressed, aborting generation--")

print()
del generator

```

**env**
- intel Ultra 7 155H igpu , driver version : 32.0.101.5972
- OS: windows11 , python3.11 , onnxruntime-dml 1.19.2 , onnxruntime-genai-directml 0.5.0.rc1 ,

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the two `onnxruntime_genai.models.builder` conversions and the supplied Python generation script, comparing CPU and DirectML results. Investigate the DirectML path and the reported environment versions; done means Qwen2.5 generation produces correct output under the DirectML execution provider.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.