microsoft / microsoft/onnxruntime-genai
0.5.2 GPU crashes if initial input is 360 zeros.
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
By setting a sequence of 360+ zeros using generatorParams.SetInputIDs(...) DirectML (C# but is not a C# related problem)
I get the following bug in 0.5.2:
```
OnnxRuntimeGenAIException: D:\a\_work\1\onnxruntime-genai\src\dml\dml_command_recorder.cpp(143)\onnxruntime-genai.dll!00007FFFD2D7AF63: (caller: 00007FFFD2D7CA85) Exception(2) tid(22ac) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.
Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess (System.IntPtr nativeResult) (at D:/a/_work/1/onnxruntime-genai/src/csharp/Result.cs:25)
Microsoft.ML.OnnxRuntimeGenAI.Generator.ComputeLogits () (at D:/a/_work/1/onnxruntime-genai/src/csharp/Generator.cs:25)
Main.Generate () (at Assets/Main.cs:202)
System.Threading.Tasks.Task.InnerInvoke () (at <9d9536d9127f4a489d989c7a566aee1c>:0)
System.Threading.Tasks.Task.Execute () (at <9d9536d9127f4a489d989c7a566aee1c>:0)
```
The sequences of zeros is just an example as it will also crash on other seemingly random prompts.
This means that GenAI is not usable in production since we cannot guarantee that a given prompt will not crash the GPU. My suggestion would be to create some tests with hundreds of initial long random token strings when testing on different GPUs. Then you should detect this bug next time.
I have no idea what could be causing this. My wild guess would be something to do with the int4 encoding/decoding as that's the main new thing recently.
Windows 10. GPU Quadro P5000.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the crash on Windows 10 with DirectML, a Quadro P5000, and 360 or more zero token IDs passed through SetInputIDs. Start at src/dml/dml_command_recorder.cpp:143 and review the call path through src/csharp/Generator.cs:25. Done means the reported prompts no longer trigger the GPU command failure, with regression coverage for long initial token sequences.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100