microsoft / microsoft/onnxruntime-genai
Bug DMLFusedNode_0_0 on second token in 0.5.2 (DML) (Wrong tensor shape)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
I updated to 0.5.2 DirectML mode. Quadro P5000 GPU. Windows. C# DirectML 1.15.4. Model: microsoft/Phi-3-mini-4k-instruct-onnx
I get the following bug. (Bug only comes up in DML mode not CPU mode. It worked in version 0.4.0).
For any prompt e.g.
"<|user|>Hello <|end|><|assistant|>"
(tokens:)
1,32010,15043,29871,32007,32001
It outputs one token but then when trying to output the second token it errors out with:
```
OnnxRuntimeGenAIException: Non-zero status code returned while running DmlFusedNode_0_0 node. Name:'DmlFusedNode_0_0' Status Message: D:\a\_work\1\s\onnxruntime\core\framework\execution_frame.cc:173 onnxruntime::IExecutionFrame::GetOrCreateNodeOutputMLValue shape && tensor.Shape() == *shape was false. OrtValue shape verification failed. Current shape:{1,32,7,96} Requested shape:{1,32,2048,96}
Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess (System.IntPtr nativeResult) (at D:/a/_work/1/onnxruntime-genai/src/csharp/Result.cs:25)
Microsoft.ML.OnnxRuntimeGenAI.Generator.ComputeLogits () (at D:/a/_work/1/onnxruntime-genai/src/csharp/Generator.cs:25)
Main.Generate () (at Assets/Main.cs:202)
```
The relevant part seems to be: Current shape:{1,32,**7**,96} Requested shape:{1,32,**2048**,96}
It appears to be not padding the tokens to the max_tokens or something.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the second-token failure with the Phi-3-mini-4k-instruct-onnx model in DirectML mode, then compare it with CPU mode and version 0.4.0. Start from the generation path shown in Assets/Main.cs:202 and Microsoft.ML.OnnxRuntimeGenAI/Generator.cs:25; done means the prompt generates past the second token without the DmlFusedNode shape mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100