microsoft / microsoft/onnxruntime-genai

Long context(input prompt length 4096 + generative output length 6000) may encounter a correctness issue for phi model in genai?

Open
#1,932 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**
Running benchmark/python/benchmark_e2e.py with `-l 4096 -g 6000 --use_prompt_set -mo` may generate chaotic outputs.

**To Reproduce**
Steps to reproduce the behavior:

- Copy Edge phi model or convert huggingface phi4 model via `python3 builder.py -m /Volumes/workspace/ort-web-perf/models/Phi-4-mini-instruct -o /Volumes/workspace/ort-web-perf/models/Phi-4-mini-instruct-onnx-origin -p int4 -e webgpu --extra_options int4_accuracy_level=4 int4_algo_config=k_quant_last`
- Running with `python3 benchmark/python/benchmark_e2e.py -i /Volumes/workspace/ort-web-perf/models/Phi-4-mini-instruct-onnx-origin -l 4096 -g 6000 --use_prompt_set -mo`

**Expected behavior**
Generate meaningful output sentences without any chaotic words.

**Screenshots**
Image

Another run:
Image

**Desktop (please complete the following information):**
- OS: MacOS
- Browser: Non
- Version: Latest main

**Additional context**
I did some other investigation and found this may be caused by the prompt, in above case, we just feed the raw text to llm without any additional conditions or requirements to llm, I then tried to add some concrete task prefix to the raw text, like `Rewrite the text with at least 6000 tokens: raw text of 4096 token length`, then the generated answers will look quite normal than before without any repetitions of sentences, see below screenshot.

output with -l 4096 -g 6000
Image

output with -l 7936 -g 6000
Image

output with -l 4096 -g 8000
Image

So I may imagine this is the capability limitations of the model itself, like you may not able to feed a raw short text to llm and let it generate a specific length of answers without additional requirements, then the tail part of tokens of the answers may fall into some uncontrollable conditions, but if you give llm a more concrete task, it may give you a quite good answer.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with benchmark/python/benchmark_e2e.py and reproduce the Phi-4 run using input lengths 4096 and 7936 with generation lengths 6000 and 8000. Compare the raw-prompt and task-prefixed outputs to determine whether the chaotic generation is a model limitation or an implementation issue. Done means the cause is established and any affected component or limitation is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.