huggingface / huggingface/diffusers

value errors in convert to/from diffusers from original stable diffusion

Offen
#11,285 11 Kommentare 2 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug stale
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

### Describe the bug

There's a hardcode somewhere for 77 tokens, when it should be using the dimensions of what is actually in the model.

I have a diffusers-layout SD1.5 model, with LongCLIP.

https://huggingface.co/opendiffusionai/xllsd-alpha0

I can pull it locally, then convert to single file format, with

python convert_diffusers_to_original_stable_diffusion.py \
--use_safetensors \
--model_path $SRCM \
--checkpoint_path $DESTM

But then if I try to convert it back, I get size errors for the text encoder not being 77 size.

I should point out that the model WORKS PROPERLY for diffusion, when loaded in diffusers format, so I dont have some funky broken model here.

### Reproduction

from transformers import CLIPTextModel, CLIPTokenizer

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
"XLLsd-phase0.safetensors",
torch_dtype=torch.float32,
use_safetensors=True)

outname = "XLLsd_recreate"
pipe.save_pretrained(outname, safe_serialization=False)

### Logs

```shell
venv/lib/python3.12/site-packages/diffusers/models/model_loading_utils.py", line 230, in load_model_dict_into_meta
raise ValueError(
ValueError: Cannot load because text_model.embeddings.position_embedding.weight expected shape torch.Size([77, 768]), but got torch.Size([248, 768]). If you want to instead overwrite randomly initialized weights, please make sure to pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example.
```

### System Info

- 🤗 Diffusers version: 0.32.2
- Platform: Linux-6.8.0-55-generic-x86_64-with-glibc2.39
- Running on Google Colab?: No
- Python version: 3.12.3
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.29.3
- Transformers version: 4.50.0
- Accelerate version: 1.5.2
- PEFT version: not installed
- Bitsandbytes version: 0.45.2
- Safetensors version: 0.5.3
- xFormers version: not installed
- Accelerator: NVIDIA GeForce RTX 4090, 24564 MiB

### Who can help?

_No response_

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start with convert_diffusers_to_original_stable_diffusion.py and trace the conversion path used by both commands, then inspect the text encoder loading path indicated by model_loading_utils.py in the error. Reproduce the reported LongCLIP round trip and confirm that conversion uses the model's actual position-embedding dimensions rather than failing with a fixed-size assumption.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
machine-learning, tooling
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.