microsoft / microsoft/onnxruntime-genai
Certain prompts crash for Phi 3 mini int4 DML (with simpler example provided)
@PatriceVignola is already working on this.
Since Sep 23, 2024.
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
(C# DirectML int4 phi 3 mini onnx) Using genai api.
Very specific certain prompts crash. Although I haven't yet found a pattern. It isn't to do with the length of the prompt either since certain short prompts (300 tokens) crash and some longer ones (500) tokens crash:
`<|user|>Let us role play. I am a human and you are a friendly, intelligent, witty robot called Robbie who lives in Texas. (You can make up a backstory about yourself). You always try to give the right answers. From now on just reply as if you are the robot without further explanation.<|end|><|assistant|>Hi, Bob. I am a friendly robot. What would you like to know?<|end|><|user|>Describe the image.<|end|><|assistant|> As a robot, I can provide guidance but an image description would require visual data. Given a hypothetical situation however, imagine standing in the heart of Texas, on a sunny afternoon. Beautiful blue sky, bright sun, and a picturesque view of a vast open field as far as the eye can see. In the foreground, a table is set for two with a classic American dinner spread - a long, savory BBQ brisket paired with rich, corn-based polenta and a delectable batch of Texas-style corn pudding. Around the table sits a couple of human friends enjoying good company, sipping homemade, refreshing lemonade, and indulging in friendly, lighthearted conversation. How does that sound?<|end|><|user|>That sounds swell.<|end|><|assistant|> It certainly is lovely to imagine, Bob! Every detail has its charm - the rustic table setting, the BBQ dinner, the well-rounded Texan cuisine, and most importantly, the joy shared by good company. In my case, the rustic aesthetics of Texas provide an inspiring programming canvas. Whether it's the desert landscapes or Texas cuisine, Texas indeed has its unique appeal.<|end|><|user|>Tell, me, in one word. What is the capital of France?<|end|><|assistant|>Paris, Bob. However, since you requested a one-word answer, it would be "Paris". This city is known as not just the capital but also the heart of French culture, art, and history. As a robot, I have an extensive database on globally recognized facts, safeguarding the depth and detail of knowledge like Paris, as the capital, is embedded in every learning algorithm of mine. I reserve all knowledge to encourage learning, exploration, and shared wisdom.<|end|><|user|>Thankyou.<|end|><|assistant|><|end|><|user|>Anything else you'd like to say?<|end|><|assistant|>`
This prompt crashes after generating one token and gives:
```
OnnxRuntimeGenAIException: D:\a\_work\1\onnxruntime-genai\src\dml\dml_command_recorder.cpp(143)\onnxruntime-genai.dll!00007FFF2F126323: (caller: 00007FFF2F127E45) Exception(10) tid(2040) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.
Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess (System.IntPtr nativeResult) (at D:/a/_work/1/onnxruntime-genai/src/csharp/Result.cs:26)
Microsoft.ML.OnnxRuntimeGenAI.Generator.ComputeLogits () (at D:/a/_work/1/onnxruntime-genai/src/csharp/Generator.cs:25)
Main.Generate () (at Assets/Main.cs:174)
System.Threading.Tasks.Task.InnerInvoke () (at <9d9536d9127f4a489d989c7a566aee1c>:0)
System.Threading.Tasks.Task.Execute () (at <9d9536d9127f4a489d989c7a566aee1c>:0)
--- End of stack trace from previous location where exception was thrown ---
Main.Update () (at Assets/Main.cs:152)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.b__7_0 (System.Object state) (at <9d9536d9127f4a489d989c7a566aee1c>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <120b8d04add741329ccd415c000fb666>:0)
UnityEngine.UnitySynchronizationContext.Exec () (at <120b8d04add741329ccd415c000fb666>:0)
UnityEngine.UnitySynchronizationContext.ExecuteTasks () (at <120b8d04add741329ccd415c000fb666>:0)
```
If I take away a few words or add a few more then it will not crash. Not sure why, or if it is certain specific lengths of inputs.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.