huggingface / huggingface/diffusers
Handling of .alpha in from-comfy converters
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
### Describe the bug
This is a question and maybe a bug report:
LoRAs in diffusers/peft-format don't support an `alpha` scaling factor. when converting another format to diffusers/peft, .alpha can be folded into lora_B]
kohya format LoRAs do hgave .alpha and for those models that diffusers implement a from-kohya converter, .alpha is supported.
For recent models, diffusers seems to have switched to supporting comfy format rather than kohya - which is mostly: diffusion_model prefix, original model keys.
And there is the issue: some from-comfy converters in diffusers have .alpha supports, others don't.
The table below is AI generated from static analysis so I cannot guarantee its correctness, but I tripped over this for Anima comfy-format LoRAs, so that one I can confirm.
And here it becomes a question: are these converters supposed to support .alpha, or do you consider comfy-format to be without alpha?
Con: comfy-format isn't really defined. it's just whatever they support, and that varies wildly between models. They do support .alpha for all models though.
Pro: it's in use. For example, the kohya-to-comfy converter that ships with kohya-ss does output .alpha
| Model | kohya import | kohya alpha | comfy import | comfy alpha |
|---|:--:|:--:|:--:|:--:|
| StableDiffusion | ✅ | ✅ | ❌ | — |
| StableDiffusionXL | ✅ | ✅ | ❌ | — |
| Flux | ✅ | ✅ | ❌ | — |
| Flux2 | ✅ | ✅ | ✅ | ❌ |
| Wan | ✅ | ✅ | ✅ | ✅ |
| Helios | ✅ | ✅ | ✅ | ✅ |
| SkyReelsV2 | ✅ | ✅ | ✅ | ✅ |
| QwenImage | ❌ | — | ✅ | ✅ |
| ZImage | ❌ | — | ✅ | ✅ |
| Ideogram4 | ❌ | — | ✅ | ✅ |
| Anima | ❌ | — | ✅ | ❌ |
| HiDreamImage | ❌ | — | ✅ | ❌ |
| HunyuanVideo | ❌ | — | ✅ | ❌ |
| LTX2 | ❌ | — | ✅ | ❌ |
| LTXVideo | ❌ | — | ✅ | ❌ |
| Lumina2 | ❌ | — | ✅ | ❌ |
| ErnieImage | ❌ | — | ✅ | ❌ |
### Reproduction
Theoretical reprodution:
Load a LoRA generated by kohya-ss, converted to comfy-format by their to-comfy converter script, load that LoRA into diffusers.
If you need actual reproduction code or a file let me know
### Logs
```shell
```
### System Info
eb0a900ca
### Who can help?
Questions on LoRA: @sayakpaul
Contributor guide
Research direction
Review the affected from-comfy converters described in the table and compare their handling with the corresponding from-kohya converters. Reproduce the reported kohya-to-comfy loading case, then establish consistent .alpha behavior across the affected models; done means the expected scaling is preserved and covered by suitable converter tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100