microsoft / microsoft/onnxruntime-genai

CodeQwen running error

Open
#805 3 comments 0 reactions 0 assignees View on GitHub
model:transformer
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**
Hello guys, much appreciate your work here.
I have some problems with running a CodeQwen model. There is no errors in converting script and the onnx models appears in the output folder. However when I am trying to launch the `model-generate.py` it throwns an Error without any really helpfull information in a traceback.
I've tried both release version and build from source, the outcome is the same.
Will appreciate any suggestions on how to debug this error properly.

**To Reproduce**
Steps to reproduce the behavior:
1. Download official CodeQwen 1.5 weights from [HF](https://huggingface.co/Qwen/CodeQwen1.5-7B)
2. `python3 onnxruntime-genai/src/python/py/models/builder.py -i models/CodeQwen1.5-7B -o models/CodeQwen1.5-7B-onnx -p fp32 -e cpu` (using the `python3 -m onnxruntime_genai.models.builder` makes no difference)
3. Run the test `python3 onnxruntime-genai/examples/python/model-generate.py -m models/CodeQwen1.5-7B-onnx -pr "# Binary search on Python" -l 64` throws an IndexError with this traceback:
```
Prompt #0: # Binary search on Python

Traceback (most recent call last):
File "/home/***/onnxruntime-genai/examples/python/model-generate.py", line 75, in
main(args)
File "/home/***/onnxruntime-genai/examples/python/model-generate.py", line 51, in main
print(tokenizer.decode(output_tokens[i]))
IndexError: map::at
```

**Desktop:**
- OS: Ubuntu 22.04

**Additional context**
Please note that using the very same commands in the same environment I was able to convert the `codegemma-1.1-2b` to the onnx format and run the test. The response was accurate (correct binary search implementation) and without any errors at all. So I am expecting there is nothing wrong with my environment.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the commands in onnxruntime-genai/src/python/py/models/builder.py and examples/python/model-generate.py on Ubuntu 22.04, then inspect the traceback at tokenizer.decode(output_tokens[i]). Compare the CodeQwen and codegemma conversion and generation results; done means the CodeQwen example provides useful failure information or completes generation without the IndexError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.