huggingface / huggingface/diffusers

PixArt Sigma PEFT LoRA loader support

Aperta
#11,216 16 commenti 0 reazioni 0 assegnatari Vedi su GitHub
contributions-welcome stale
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

**Is your feature request related to a problem? Please describe.**

Currently, the upstream PixArt trainer does this:

```py

transformer = get_peft_model(transformer, lora_config)
if args.mixed_precision == "fp16":
# only upcast trainable parameters (LoRA) into fp32
cast_training_params(transformer, dtype=torch.float32)

transformer.print_trainable_parameters()

# 10. Handle saving and loading of checkpoints
# `accelerate` 0.16.0 will have better support for customized saving
if version.parse(accelerate.__version__) >= version.parse("0.16.0"):
# create custom saving & loading hooks so that `accelerator.save_state(...)` serializes in a nice format
def save_model_hook(models, weights, output_dir):
if accelerator.is_main_process:
transformer_ = accelerator.unwrap_model(transformer)
lora_state_dict = get_peft_model_state_dict(transformer_, adapter_name="default")
StableDiffusionPipeline.save_lora_weights(os.path.join(output_dir, "transformer_lora"), lora_state_dict)
# save weights in peft format to be able to load them back
transformer_.save_pretrained(output_dir)

for _, model in enumerate(models):
# make sure to pop weight so that corresponding model is not saved again
weights.pop()
```

**Describe the solution you'd like.**

I would like for the PixArtSigmaPipeline to have the necessary mix-ins and inference support instead.

**Describe alternatives you've considered.**

I have considered avoiding the pipeline methods and using workarounds like upstream does, but instead I would prefer consistency for user experience.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start by reading the PixArtSigmaPipeline and the upstream PixArt trainer flow shown in the issue, especially its PEFT LoRA setup and checkpoint hooks. Determine the pipeline mix-ins and inference behavior needed for loading PixArt Sigma LoRA weights, then verify that the pipeline provides a consistent alternative to the trainer workarounds.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
48/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.