microsoft / microsoft/onnxruntime-genai
0.5.2 DML 2x to 4x Slower than 0.4.0 (Big regression)
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
genai 0.5.2 appears to be between 2x and 4x slower than genai 0.4.0, in fact it is only about 50% faster than CPU mode.
If have just tested both 0.4.0 and 0.5.2 and it definitely is a vast difference in DML mode.
GPU Quadro P5000.
model: microsoft/Phi-3-mini-4k-instruct-onnx
c#
BTW, I checked it is not the new DirectML.dll library causing this.
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file, test, or entry point is named. First reproduce the comparison between genai 0.4.0 and 0.5.2 in DML mode with the Phi-3-mini-4k-instruct-onnx model on a Quadro P5000, then compare against CPU mode. Done means identifying the regression and validating improved DML performance without attributing it to the new DirectML.dll library.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, csharp
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100