huggingface / huggingface/diffusers

PixArt Sigma PEFT LoRA loader support

Open
#11,216 16 comments 0 reactions 0 assignees View on GitHub
contributions-welcome stale
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

**Is your feature request related to a problem? Please describe.**

Currently, the upstream PixArt trainer does this:

```py

transformer = get_peft_model(transformer, lora_config)
if args.mixed_precision == "fp16":
# only upcast trainable parameters (LoRA) into fp32
cast_training_params(transformer, dtype=torch.float32)

transformer.print_trainable_parameters()

# 10. Handle saving and loading of checkpoints
# `accelerate` 0.16.0 will have better support for customized saving
if version.parse(accelerate.__version__) >= version.parse("0.16.0"):
# create custom saving & loading hooks so that `accelerator.save_state(...)` serializes in a nice format
def save_model_hook(models, weights, output_dir):
if accelerator.is_main_process:
transformer_ = accelerator.unwrap_model(transformer)
lora_state_dict = get_peft_model_state_dict(transformer_, adapter_name="default")
StableDiffusionPipeline.save_lora_weights(os.path.join(output_dir, "transformer_lora"), lora_state_dict)
# save weights in peft format to be able to load them back
transformer_.save_pretrained(output_dir)

for _, model in enumerate(models):
# make sure to pop weight so that corresponding model is not saved again
weights.pop()
```

**Describe the solution you'd like.**

I would like for the PixArtSigmaPipeline to have the necessary mix-ins and inference support instead.

**Describe alternatives you've considered.**

I have considered avoiding the pipeline methods and using workarounds like upstream does, but instead I would prefer consistency for user experience.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.