huggingface / huggingface/diffusers

Handling of .alpha in from-comfy converters

Open
#14,088 8 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Describe the bug

This is a question and maybe a bug report:
LoRAs in diffusers/peft-format don't support an `alpha` scaling factor. when converting another format to diffusers/peft, .alpha can be folded into lora_B]

kohya format LoRAs do hgave .alpha and for those models that diffusers implement a from-kohya converter, .alpha is supported.

For recent models, diffusers seems to have switched to supporting comfy format rather than kohya - which is mostly: diffusion_model prefix, original model keys.
And there is the issue: some from-comfy converters in diffusers have .alpha supports, others don't.

The table below is AI generated from static analysis so I cannot guarantee its correctness, but I tripped over this for Anima comfy-format LoRAs, so that one I can confirm.

And here it becomes a question: are these converters supposed to support .alpha, or do you consider comfy-format to be without alpha?
Con: comfy-format isn't really defined. it's just whatever they support, and that varies wildly between models. They do support .alpha for all models though.
Pro: it's in use. For example, the kohya-to-comfy converter that ships with kohya-ss does output .alpha

| Model | kohya import | kohya alpha | comfy import | comfy alpha |
|---|:--:|:--:|:--:|:--:|
| StableDiffusion | ✅ | ✅ | ❌ | — |
| StableDiffusionXL | ✅ | ✅ | ❌ | — |
| Flux | ✅ | ✅ | ❌ | — |
| Flux2 | ✅ | ✅ | ✅ | ❌ |
| Wan | ✅ | ✅ | ✅ | ✅ |
| Helios | ✅ | ✅ | ✅ | ✅ |
| SkyReelsV2 | ✅ | ✅ | ✅ | ✅ |
| QwenImage | ❌ | — | ✅ | ✅ |
| ZImage | ❌ | — | ✅ | ✅ |
| Ideogram4 | ❌ | — | ✅ | ✅ |
| Anima | ❌ | — | ✅ | ❌ |
| HiDreamImage | ❌ | — | ✅ | ❌ |
| HunyuanVideo | ❌ | — | ✅ | ❌ |
| LTX2 | ❌ | — | ✅ | ❌ |
| LTXVideo | ❌ | — | ✅ | ❌ |
| Lumina2 | ❌ | — | ✅ | ❌ |
| ErnieImage | ❌ | — | ✅ | ❌ |

### Reproduction

Theoretical reprodution:
Load a LoRA generated by kohya-ss, converted to comfy-format by their to-comfy converter script, load that LoRA into diffusers.

If you need actual reproduction code or a file let me know

### Logs

```shell

```

### System Info

eb0a900ca

### Who can help?

Questions on LoRA: @sayakpaul

Contributor guide

Open the contributing guide

Research direction

Review the affected from-comfy converters described in the table and compare their handling with the corresponding from-kohya converters. Reproduce the reported kohya-to-comfy loading case, then establish consistent .alpha behavior across the affected models; done means the expected scaling is preserved and covered by suitable converter tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.