microsoft / microsoft/onnxruntime

[Feature Request] Adapters DML support

Open
#23,503 1 comment 0 reactions 1 assignee Claimed by @crahrig View on GitHub
ep:DML feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

I've made issues on [olive](https://github.com/microsoft/Olive/issues/1544) and [genai](https://github.com/microsoft/onnxruntime-genai/issues/1181) but having looked into it more deeply I think this issue belongs here. There seems to have been [this PR](https://github.com/microsoft/onnxruntime/pull/22396) attempt to give adapter support to DML which was then reverted in [this PR](https://github.com/microsoft/onnxruntime/pull/22814). I have tried this solution (built from source) but I get the following error when calling `LoadAdapter` with a valid `.onnx_adapter` path (generated with olive):
`System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.'`
This has been replaced with the following (confusing) error message instead which is what you get when using more recent builds:
`Data transfer is not available for the specified device allocator, it also must not be a CPU allocator`
Is this to do with DataTransfer being broken in DML? Because [this branch](https://github.com/microsoft/onnxruntime/tree/user/pavignol/fix-dml-data-transfer) worked on fixes for that that were never incorporated, or did they never work to begin with?

I can see that [this issue](https://github.com/microsoft/onnxruntime/issues/21483) regarding 0 dim tensors being unsupported by DML, this has been fixed in CUDA with [this PR](https://github.com/microsoft/onnxruntime/pull/21525) but no fix was given to DML. I mention because the 0 dim system is how adapters are currently implemented and:

> Yeah, that's illegal from the DirectML API validator point of view, multiplying nothing times nothing and expecting something 😉.
_Originally posted by @fdwr in [#21483](https://github.com/microsoft/onnxruntime/issues/21483#issuecomment-2249543532)_

If this is the source of the issue, is it not possible to have an alternative where the specific dim is defined (i.e. a set LoRA rank) which all adapters use? This gives means you can't use LoRA adapters of different sizes with one model but honestly that's not really much of an issue in comparison to just not being able to use the Adapters API with DML at all.

Is there any update on whether Adapters support in DML is a priority or when we might see it? I think it's misleading that ORT has these post about [multilora support](https://onnxruntime.ai/blogs/multilora), [DML for edge deployment](https://onnxruntime.ai/blogs/accelerating-phi-3-small-medium) and [olive DML support](https://devblogs.microsoft.com/windowsai/optimize-directml-performance-with-olive/) but don't have it specified anywhere that LoRA with DML doesn't work at all.

### Describe scenario use case

Multi-LoRA serving on edge hardware with DirectML Execution Provider.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.