microsoft / microsoft/onnxruntime-genai

Bug DMLFusedNode_0_0 on second token in 0.5.2 (DML) (Wrong tensor shape)

Open
#1,112 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ep:DML
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

I updated to 0.5.2 DirectML mode. Quadro P5000 GPU. Windows. C# DirectML 1.15.4. Model: microsoft/Phi-3-mini-4k-instruct-onnx

I get the following bug. (Bug only comes up in DML mode not CPU mode. It worked in version 0.4.0).

For any prompt e.g.
"<|user|>Hello <|end|><|assistant|>"
(tokens:)
1,32010,15043,29871,32007,32001

It outputs one token but then when trying to output the second token it errors out with:

```
OnnxRuntimeGenAIException: Non-zero status code returned while running DmlFusedNode_0_0 node. Name:'DmlFusedNode_0_0' Status Message: D:\a\_work\1\s\onnxruntime\core\framework\execution_frame.cc:173 onnxruntime::IExecutionFrame::GetOrCreateNodeOutputMLValue shape && tensor.Shape() == *shape was false. OrtValue shape verification failed. Current shape:{1,32,7,96} Requested shape:{1,32,2048,96}

Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess (System.IntPtr nativeResult) (at D:/a/_work/1/onnxruntime-genai/src/csharp/Result.cs:25)
Microsoft.ML.OnnxRuntimeGenAI.Generator.ComputeLogits () (at D:/a/_work/1/onnxruntime-genai/src/csharp/Generator.cs:25)
Main.Generate () (at Assets/Main.cs:202)
```

The relevant part seems to be: Current shape:{1,32,**7**,96} Requested shape:{1,32,**2048**,96}

It appears to be not padding the tokens to the max_tokens or something.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the second-token failure with the Phi-3-mini-4k-instruct-onnx model in DirectML mode, then compare it with CPU mode and version 0.4.0. Start from the generation path shown in Assets/Main.cs:202 and Microsoft.ML.OnnxRuntimeGenAI/Generator.cs:25; done means the prompt generates past the second token without the DmlFusedNode shape mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.