microsoft / microsoft/onnxruntime

OpenCL and Mali GPU support left out of all execution providers

Open
#20,896 3 comments 1 reaction 0 assignees View on GitHub
ep:ACL ep:ArmNN ep:tvm feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

I was trying to migrate from MLC-LLM to onnxruntime to run Phi-3 on an Orange Pi 5 but I realize that among ALL your execution providers there isn't a single one that takes advantage of the GPU or NPU!

Such a shame. You specifically develop support for ARM ACL and ARMNN, both which specifically support Mali GPU, but you only provide support for CPU (???).

Then you support the RKNN framework...but only for a single model of the chip.

Then you support TVM in preview, but only relay not relax, so can't be used for LLMs.

Please! The adoption of onnxruntime for LLMs is on the line.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing ONNX Runtime's execution providers and the mentioned ARM ACL, ARMNN, RKNN, and TVM integrations. Determine the scope for supporting OpenCL or Mali GPU/NPU execution on the Orange Pi 5, then define completion as a working provider path for the requested hardware and Phi-3 workloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
embedded-iot, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.