microsoft / microsoft/onnxruntime
OpenCL and Mali GPU support left out of all execution providers
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
I was trying to migrate from MLC-LLM to onnxruntime to run Phi-3 on an Orange Pi 5 but I realize that among ALL your execution providers there isn't a single one that takes advantage of the GPU or NPU!
Such a shame. You specifically develop support for ARM ACL and ARMNN, both which specifically support Mali GPU, but you only provide support for CPU (???).
Then you support the RKNN framework...but only for a single model of the chip.
Then you support TVM in preview, but only relay not relax, so can't be used for LLMs.
Please! The adoption of onnxruntime for LLMs is on the line.
Contributor guide
Research direction
Start by reviewing ONNX Runtime's execution providers and the mentioned ARM ACL, ARMNN, RKNN, and TVM integrations. Determine the scope for supporting OpenCL or Mali GPU/NPU execution on the Orange Pi 5, then define completion as a working provider path for the requested hardware and Phi-3 workloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- embedded-iot, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100