bytecodealliance / bytecodealliance/wasmtime
Support additional Execution Providers in ONNX `wasi-nn` backend
- Dominant language
- Rust
- Stars
- 18.6k
- Forks
- 1.8k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 121
Description
#### Feature
Currently the ONNX backend in `wasmtime-wasi-nn` only uses the default CPU execution provider and ignores the `ExecutionTarget` requested by the WASM caller.
https://github.com/bytecodealliance/wasmtime/blob/24c1388cd74ab321d60af147fc074d12166258fd/crates/wasi-nn/src/backend/onnxruntime.rs#L21-L33
I would like to suggest adding support for additional execution providers (CUDA, TensorRT, ROCm, ...) to `wasmtime-wasi-nn`.
#### Benefit
Improved performance for WASM modules using the `wasi-nn` API.
#### Implementation
`ort` already has support for many execution providers, so integrating these into `wasmtime-wasi-nn` should not be to much work.
I would be interested in looking into this, however, I only really have the means to test the DirectML and NVIDIA CUDA / TensorRT EPs.
#### Alternatives
Leave it to the users to add support for additional execution providers.
Contributor guide
Assessment
This issue has not been assessed yet.