invoke-ai / invoke-ai/InvokeAI

[enhancement]: Optional acceleration for non-distilled models

Open
#9,547 0 comments 0 reactions 1 assignee Claimed by @dunkeroni View on GitHub
enhancement
Dominant language
Python
Stars
28.2k
Forks
3k
Avg merge
6d 5h
Merged PRs (30d)
19

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Contact Details

_No response_

### What should this feature add?

It may be helpful to add support for easy access to distillation LoRAs on non-distilled architectures, e.g. for Strix Halo users and those with lower-end cards that have sufficient memory but slower rendering speeds. I reckon that this would be best done in stages:

1. Testing support for acceleration LoRAs (noting https://github.com/invoke-ai/InvokeAI/issues/8415)
2. Adding relevant LoRAs to the Starter Models list
3. Adding a checkbox to the linear UI that can be activated by users when desired
4. Guarding against incompatible CFG settings when acceleration is activated

Here is a list of likely LoRAs for inclusion:

| Architecture | Steps | Huggingface Repository |
|--------------------|----------|------------------------------------|
| Stable Diffusion 1.5 | 8 (CFG preserved)| [ByteDance/Hyper-SD](https://huggingface.co/ByteDance/Hyper-SD/blob/main/Hyper-SD15-8steps-CFG-lora.safetensors) |
| Stable Diffusion XL | 8 (CFG preserved) | [ByteDance/Hyper-SD](https://huggingface.co/ByteDance/Hyper-SD/blob/main/Hyper-SDXL-8steps-CFG-lora.safetensors) |
| Stable Diffusion 3.5 Medium | 4 (requires CFG=1) | [tensorart/stable-diffusion-3.5-medium-turbo](https://huggingface.co/tensorart/stable-diffusion-3.5-medium-turbo/blob/main/lora_sd3.5m_4steps.safetensors)
| Flux.1 Dev | 4* | [Lingyuzhou/Hyper_Flux.1_Dev_4_step_Lora](https://huggingface.co/Lingyuzhou/Hyper_Flux.1_Dev_4_step_Lora/blob/main/Hyper-Flux.1-Dev%204-step-Lora.safetensors) |
| Anima | 4 (requires CFG=1) | [sorryhyun/anima-turbo-4step](https://huggingface.co/sorryhyun/anima-turbo-4step/blob/main/anima_turbo_4step_v2_longer.safetensors) |
| Krea 2 Turbo | 4* | [lvladikov/Krea2-Turbo-Distill-4step-LoRA](https://huggingface.co/lvladikov/Krea2-Turbo-Distill-4step-LoRA/blob/main/krea2_turbo_4step_rank_64_lora.safetensors) |

\* Base model already distilled

### Alternatives

The CFG-preserved LoRAs listed usually have lower-step alternatives that require the CFG to be set to 1. I have opted here for the more conservative options.

### Additional Content

- Fooocus previously [bundled](https://github.com/mashb1t/Fooocus/blob/21a44c214a574536216e03209ee034f35c6527c9/update_log.md#240) the 4-step LoRA with its SDXL implementation, making it available via a preset
- Stable Diffusion 3.5 Large has a [4-step Turbo model](https://huggingface.co/stabilityai/stable-diffusion-3.5-large-turbo) rather than an acceleration LoRA (but note #9399)
- If this FR is implemented, it may be possible to support CPU-only generation on select architectures using 1-step LoRAs, as demonstrated in [this](https://github.com/rupeshs/fastsdcpu) existing Diffusers implementation
- [SD1.5](https://huggingface.co/ByteDance/Hyper-SD/blob/main/Hyper-SD15-1step-lora.safetensors)
- [SDXL](https://huggingface.co/ByteDance/Hyper-SD/blob/main/Hyper-SDXL-1step-lora.safetensors)
- [Flux.2 Klein 4B](https://huggingface.co/epfl-vita/flux2-klein-1step-rdm)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.