microsoft / microsoft/onnxruntime-genai

_com_error at memory location 0x0000006A74D6D680 when calling generator.ComputeLogits()

Open
#886 4 comments 0 reactions 1 assignee Claimed by @PatriceVignola View on GitHub
ep:DML model:transformer platform:windows
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**
I've tried to copy the HelloPhi example for loading and calling phi-3.5 in DirectML for my application. When I get to the [generator.ComputeLogits()](https://github.com/microsoft/onnxruntime-genai/blob/main/examples/csharp/HelloPhi/Program.cs#L101) part, I get a ton (e.g. hundreds) of the following memory errors:

```
Exception thrown at 0x00007FFFF9ED6D9A in .exe: Microsoft C++ exception: _com_error at memory location 0x0000006A74D6D680.
```

After a few minutes, the inference engine starts generating the text.

**To Reproduce**
Steps to reproduce the behavior:
1. Copy the DirectML version of the HelloPhi example as a new `Blank App, Packaged with Windows Application Packing Project (WinUI 3 in Desktop)` app with framework net8.0-windows10.0.19041.0 (n.b. I'm using WindowsSdkPackageVersion 10.0.19041.38).
2. Download and convert phi-3.5 to DirectML (or rely on the onnx dml verison at microsoft/Phi-3-mini-128k-instruct-onnx on huggingface - I've tried both models).
3. Run the app with the downloaded model (making sure to break when c++ exceptions are thrown.
4. See error

**Expected behavior**
No memory errors to show and to start generating text immediately (rather than after 3-5 minutes of the memory exceptions).

**Screenshots**
![image](https://github.com/user-attachments/assets/1977fa70-8e5f-4186-bc44-2cbf904dc41d)
![image](https://github.com/user-attachments/assets/a9298764-2cde-43a9-9b3f-5915b217fee0)
![image](https://github.com/user-attachments/assets/aceef580-f902-4331-9c04-f294afe92800)

**Desktop (please complete the following information):**
- OS: Windows 11

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.