microsoft / microsoft/onnxruntime
[Feature Request] Attention slicing?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
As per this article..
I put
pipeline.enable_attention_slicing("max")
in the torch conversion file to convert the standard diffusion model on onnx format.
This resulted in a modified UNET onnx file, with the results:
attention slicing OFF : 8.6GB VRAM 0.7 seconds per step, session loading in 8 seconds
attention slicing ON: 7.1GB VRAM 1 seconds per step, session loading in 50 seconds
(The VRAM includes other things as well)
As you can see the attention slicing saved on about 1.5GB of VRAM. Good for lower end GPUs.
But for some reason it took 6 times longer to load the session file. (Is it somehow making the optimization really slow? why?)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the torch conversion file and reproduce the two configurations using pipeline.enable_attention_slicing("max"). Compare ONNX session loading time, per-step latency, and VRAM usage, then inspect the relevant ONNX Runtime optimization or session-loading path. Done means the loading regression is explained and a supported direction for attention slicing is documented or defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pytorch
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100