microsoft / microsoft/onnxruntime-genai
Error running phi-3 vision directml P5000 gpu
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
I am running the [phi3 vision directml](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct-onnx-directml/tree/main/directml-int4-rtn-block-32) tutorial [code](https://onnxruntime.ai/docs/genai/tutorials/phi3-v.html) on NVidia Quadro P5000 GPU, 16GB VRAM, +12GB RAM (Windows 10) , but it fails when I try to put an image path in there:

It works without putting an image there.

I have tried both jpg and png images. Here is my image:

Any ideas what could be wrong?
I have 16GB GPU RAM and 12GB RAM and it's only using about half of it so I don't think that's the problem.
Come to think of it the phi-3 vision tutorial doesn't say it supports DML yet... even though there is a DML model. It says "Support for DirectML is coming soon!" But not sure how soon this means.
I tried it [in C#](https://github.com/microsoft/onnxruntime-genai/tree/main/examples/csharp/HelloPhi3V) and get the same error ☹
```
OnnxRuntimeGenAIException: Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_5' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2557)\onnxruntime.dll!00007FF8171EFC45: (caller: 00007FF81780254D) Exception(9) tid(5324) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.
Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess (System.IntPtr nativeResult) (at D:/a/_work/1/onnxruntime-genai/src/csharp/Result.cs:26)
Microsoft.ML.OnnxRuntimeGenAI.Generator.ComputeLogits () (at D:/a/_work/1/onnxruntime-genai/src/csharp/Generator.cs:25)
```
I feel like my specifications meet above the recommended.
(I also tried it with the CPU only version and it works but is incredibly slow. e.g. 5 minutes+ to get a response even with a very small image. The image size doesn't seem to make a difference which is odd(!) I'm not sure how the vision thing works. Is it iterating over every small patch or something?).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.