microsoft / microsoft/onnxruntime-genai

Error running phi-3 vision directml P5000 gpu

Open
#822 3 comments 0 reactions 1 assignee Claimed by @PatriceVignola View on GitHub
ep:DML model:transformer platform:windows
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

I am running the [phi3 vision directml](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct-onnx-directml/tree/main/directml-int4-rtn-block-32) tutorial [code](https://onnxruntime.ai/docs/genai/tutorials/phi3-v.html) on NVidia Quadro P5000 GPU, 16GB VRAM, +12GB RAM (Windows 10) , but it fails when I try to put an image path in there:

![image](https://github.com/user-attachments/assets/cc96232d-f543-40f6-9fc5-892cc9ff1975)

It works without putting an image there.

![image](https://github.com/user-attachments/assets/178bdfef-8cca-4933-b5ae-23321c7523af)

I have tried both jpg and png images. Here is my image:
![decoded](https://github.com/user-attachments/assets/8f6c57d1-dd6d-47be-8dd2-e2ec2183e2f2)

Any ideas what could be wrong?

I have 16GB GPU RAM and 12GB RAM and it's only using about half of it so I don't think that's the problem.

Come to think of it the phi-3 vision tutorial doesn't say it supports DML yet... even though there is a DML model. It says "Support for DirectML is coming soon!" But not sure how soon this means.

I tried it [in C#](https://github.com/microsoft/onnxruntime-genai/tree/main/examples/csharp/HelloPhi3V) and get the same error ☹

```
OnnxRuntimeGenAIException: Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_5' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2557)\onnxruntime.dll!00007FF8171EFC45: (caller: 00007FF81780254D) Exception(9) tid(5324) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.

Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess (System.IntPtr nativeResult) (at D:/a/_work/1/onnxruntime-genai/src/csharp/Result.cs:26)
Microsoft.ML.OnnxRuntimeGenAI.Generator.ComputeLogits () (at D:/a/_work/1/onnxruntime-genai/src/csharp/Generator.cs:25)
```
I feel like my specifications meet above the recommended.
(I also tried it with the CPU only version and it works but is incredibly slow. e.g. 5 minutes+ to get a response even with a very small image. The image size doesn't seem to make a difference which is odd(!) I'm not sure how the vision thing works. Is it iterating over every small patch or something?).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.