microsoft / microsoft/onnxruntime-genai

0.5.2 GPU crashes if initial input is 360 zeros.

Open
#1,113 0 comments 0 reactions 0 assignees View on GitHub
crash ep:DML
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

By setting a sequence of 360+ zeros using generatorParams.SetInputIDs(...) DirectML (C# but is not a C# related problem)

I get the following bug in 0.5.2:

```
OnnxRuntimeGenAIException: D:\a\_work\1\onnxruntime-genai\src\dml\dml_command_recorder.cpp(143)\onnxruntime-genai.dll!00007FFFD2D7AF63: (caller: 00007FFFD2D7CA85) Exception(2) tid(22ac) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.

Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess (System.IntPtr nativeResult) (at D:/a/_work/1/onnxruntime-genai/src/csharp/Result.cs:25)
Microsoft.ML.OnnxRuntimeGenAI.Generator.ComputeLogits () (at D:/a/_work/1/onnxruntime-genai/src/csharp/Generator.cs:25)
Main.Generate () (at Assets/Main.cs:202)
System.Threading.Tasks.Task.InnerInvoke () (at <9d9536d9127f4a489d989c7a566aee1c>:0)
System.Threading.Tasks.Task.Execute () (at <9d9536d9127f4a489d989c7a566aee1c>:0)
```

The sequences of zeros is just an example as it will also crash on other seemingly random prompts.

This means that GenAI is not usable in production since we cannot guarantee that a given prompt will not crash the GPU. My suggestion would be to create some tests with hundreds of initial long random token strings when testing on different GPUs. Then you should detect this bug next time.

I have no idea what could be causing this. My wild guess would be something to do with the int4 encoding/decoding as that's the main new thing recently.

Windows 10. GPU Quadro P5000.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the crash on Windows 10 with DirectML, a Quadro P5000, and 360 or more zero token IDs passed through SetInputIDs. Start at src/dml/dml_command_recorder.cpp:143 and review the call path through src/csharp/Generator.cs:25. Done means the reported prompts no longer trigger the GPU command failure, with regression coverage for long initial token sequences.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.