bytecodealliance / bytecodealliance/wasmtime

Support additional Execution Providers in ONNX `wasi-nn` backend

Open
#8,547 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
18.6k
Forks
1.8k
Avg merge
1d 19h
Merged PRs (30d)
121

Description

#### Feature
Currently the ONNX backend in `wasmtime-wasi-nn` only uses the default CPU execution provider and ignores the `ExecutionTarget` requested by the WASM caller.
https://github.com/bytecodealliance/wasmtime/blob/24c1388cd74ab321d60af147fc074d12166258fd/crates/wasi-nn/src/backend/onnxruntime.rs#L21-L33

I would like to suggest adding support for additional execution providers (CUDA, TensorRT, ROCm, ...) to `wasmtime-wasi-nn`.

#### Benefit
Improved performance for WASM modules using the `wasi-nn` API.

#### Implementation
`ort` already has support for many execution providers, so integrating these into `wasmtime-wasi-nn` should not be to much work.
I would be interested in looking into this, however, I only really have the means to test the DirectML and NVIDIA CUDA / TensorRT EPs.

#### Alternatives
Leave it to the users to add support for additional execution providers.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.