huggingface / huggingface/diffusers
Handling of .alpha in from-comfy converters
Dieses Issue hat noch niemand übernommen.
- Vorherrschende Sprache
- Python
- Sterne
- 34.5k
- Forks
- 7.3k
- Ø Merge
- 3 T. 3 Std.
- Gemergte PRs (30 T.)
- 91
Beschreibung
Describe the bug
This is a question and maybe a bug report:
LoRAs in diffusers/peft-format don't support an alpha scaling factor. when converting another format to diffusers/peft, .alpha can be folded into lora_B]
kohya format LoRAs do hgave .alpha and for those models that diffusers implement a from-kohya converter, .alpha is supported.
For recent models, diffusers seems to have switched to supporting comfy format rather than kohya - which is mostly: diffusion_model prefix, original model keys.
And there is the issue: some from-comfy converters in diffusers have .alpha supports, others don't.
The table below is AI generated from static analysis so I cannot guarantee its correctness, but I tripped over this for Anima comfy-format LoRAs, so that one I can confirm.
And here it becomes a question: are these converters supposed to support .alpha, or do you consider comfy-format to be without alpha?
Con: comfy-format isn't really defined. it's just whatever they support, and that varies wildly between models. They do support .alpha for all models though.
Pro: it's in use. For example, the kohya-to-comfy converter that ships with kohya-ss does output .alpha
| Model | kohya import | kohya alpha | comfy import | comfy alpha |
|---|---|---|---|---|
| StableDiffusion | ✅ | ✅ | ❌ | — |
| StableDiffusionXL | ✅ | ✅ | ❌ | — |
| Flux | ✅ | ✅ | ❌ | — |
| Flux2 | ✅ | ✅ | ✅ | ❌ |
| Wan | ✅ | ✅ | ✅ | ✅ |
| Helios | ✅ | ✅ | ✅ | ✅ |
| SkyReelsV2 | ✅ | ✅ | ✅ | ✅ |
| QwenImage | ❌ | — | ✅ | ✅ |
| ZImage | ❌ | — | ✅ | ✅ |
| Ideogram4 | ❌ | — | ✅ | ✅ |
| Anima | ❌ | — | ✅ | ❌ |
| HiDreamImage | ❌ | — | ✅ | ❌ |
| HunyuanVideo | ❌ | — | ✅ | ❌ |
| LTX2 | ❌ | — | ✅ | ❌ |
| LTXVideo | ❌ | — | ✅ | ❌ |
| Lumina2 | ❌ | — | ✅ | ❌ |
| ErnieImage | ❌ | — | ✅ | ❌ |
Reproduction
Theoretical reprodution:
Load a LoRA generated by kohya-ss, converted to comfy-format by their to-comfy converter script, load that LoRA into diffusers.
If you need actual reproduction code or a file let me know
Logs
System Info
eb0a900ca
Who can help?
Questions on LoRA: @sayakpaul
Beitragsleitfaden
Erste Schritte
- Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
- Forke das Repository und arbeite in einem Branch.
- Öffne einen Pull Request, der die Issue-Nummer nennt.
Rechercherichtung
Überprüfe die in der Tabelle beschriebenen betroffenen from-comfy-Konverter und vergleiche deren Verarbeitung mit den entsprechenden from-kohya-Konvertern. Reproduziere den gemeldeten kohya-to-comfy-Ladefall und stelle anschließend ein konsistentes .alpha-Verhalten über die betroffenen Modelle hinweg her; abgeschlossen ist die Aufgabe, wenn die erwartete Skalierung erhalten bleibt und durch geeignete Konvertertests abgedeckt ist.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python, pytorch
- Bereich
- machine-learning
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 48/100