huggingface / huggingface/diffusers
DiffusionPipeline.download() on Windows silently skips all subfolder config.json files with huggingface_hub>=1.22.0
Nadie ha tomado este issue todavía.
- Lenguaje dominante
- Python
- Estrellas
- 34.5k
- Forks
- 7.3k
- Merge medio
- 3 d 3 h
- PR fusionados (30 d)
- 91
Descripción
Describe the bug
On Windows, DiffusionPipeline.download() builds the allow patterns for per-component configs with os.path.join:
https://github.com/huggingface/diffusers/blob/b8905b9b0f01/src/diffusers/pipelines/pipeline_utils.py#L1740-L1741
which produces vae\config.json, while hub repo paths always use forward slashes (vae/config.json).
Until recently this was harmless: filter_repo_objects in huggingface_hub matched patterns with fnmatch.fnmatch, which on Windows runs os.path.normcase on both pattern and path and unifies the separators, so the backslash pattern still matched. huggingface_hub 1.22.0 switched it to fnmatch.fnmatchcase to make matching case-sensitive on all platforms (huggingface/huggingface_hub#4435), and fnmatchcase does no normalization. Since that release the backslash patterns match nothing:
>>> import os
>>> from fnmatch import fnmatchcase
>>> os.path.join("vae", "config.json")
'vae\\config.json'
>>> fnmatchcase("vae/config.json", os.path.join("vae", "config.json"))
False # True on Linux/macOS, and True on Windows with fnmatch.fnmatch (hub <= 1.21.0)
Reproduction
Run on Windows with huggingface_hub>=1.22.0:
import os
from diffusers import DiffusionPipeline
folder = DiffusionPipeline.download("hf-internal-testing/tiny-stable-diffusion-torch", cache_dir="./fresh-cache")
for root, _, files in os.walk(folder):
for f in files:
print(os.path.relpath(os.path.join(root, f), folder).replace(os.sep, "/"))
Output: weights are fetched, but text_encoder/config.json, unet/config.json and vae/config.json are missing:
model_index.json
feature_extractor/preprocessor_config.json
scheduler/scheduler_config.json
text_encoder/pytorch_model.bin
tokenizer/merges.txt
tokenizer/special_tokens_map.json
tokenizer/tokenizer_config.json
tokenizer/vocab.json
unet/diffusion_pytorch_model.bin
vae/diffusion_pytorch_model.bin
DiffusionPipeline.from_pretrained("hf-internal-testing/tiny-stable-diffusion-torch", cache_dir="./fresh-cache") then fails, and keeps failing on every retry because the incomplete snapshot is considered fully cached. The same snippet on Linux, or on Windows with huggingface_hub<=1.21.0, also fetches the three config.json files.
Logs
`from_pretrained` failure with the reproduction above (transformers 5.x text encoder hits the missing config first and instantiates a default config, so the error is a shape mismatch):
File "issue_repro.py", line 3, in <module>
pipe = DiffusionPipeline.from_pretrained("hf-internal-testing/tiny-stable-diffusion-torch", cache_dir="./fresh-cache")
File "...\site-packages\diffusers\pipelines\pipeline_utils.py", line 1057, in from_pretrained
loaded_sub_model = load_sub_model(
File "...\site-packages\diffusers\pipelines\pipeline_loading_utils.py", line 910, in load_sub_model
loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
File "...\site-packages\transformers\modeling_utils.py", line 4369, in from_pretrained
loading_info = cls._finalize_model_loading(model, load_config, loading_info)
File "...\site-packages\transformers\modeling_utils.py", line 4545, in _finalize_model_loading
log_state_dict_report(
File "...\site-packages\transformers\utils\loading_report.py", line 278, in log_state_dict_report
raise RuntimeError(
RuntimeError: You set `ignore_mismatched_sizes` to `False`, thus raising an error. For details look at the above report!
When a diffusers-native component is the first one loaded from the cached folder (the original real-world case, a pipeline loaded with `transformer` and `text_encoder` passed in, so the VAE came from the snapshot):
File "...\site-packages\diffusers\pipelines\pipeline_utils.py", line 1057, in from_pretrained
loaded_sub_model = load_sub_model(
File "...\site-packages\diffusers\pipelines\pipeline_loading_utils.py", line 910, in load_sub_model
loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
File "...\site-packages\diffusers\models\modeling_utils.py", line 1127, in from_pretrained
config, unused_kwargs, commit_hash = cls.load_config(
File "...\site-packages\diffusers\configuration_utils.py", line 418, in load_config
raise EnvironmentError(
OSError: Error no file named config.json found in directory ...\models--vladmandic--Krea-2-Base-sdnq-hadamard-uint4\snapshots\c8ad74b656917d746124cb1aa3991c93a227c7dd\vae.
System Info
- 🤗 Diffusers version: 0.39.0.dev0
- Platform: Windows-11-10.0.26200-SP0
- Running on Google Colab?: No
- Python version: 3.13.11
- PyTorch version (GPU?): 2.12.0+cu132 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 1.22.0
- Transformers version: 5.14.0.dev0
- Accelerate version: 1.14.0
- PEFT version: 0.19.1
- Safetensors version: 0.8.0
- xFormers version: not installed
- Accelerator: NVIDIA RTX 2000 Ada Generation Laptop GPU, 8188 MiB
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
Who can help?
@sayakpaul @DN6
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Línea de trabajo
Comienza en src/diffusers/pipelines/pipeline_utils.py alrededor de las líneas 1740-1741, donde DiffusionPipeline.download() construye los patrones allow de la configuración de componentes. Reproduce la descarga en Windows con huggingface_hub 1.22.0 usando el ejemplo tiny-stable-diffusion-torch y verifica después que text_encoder/config.json, unet/config.json y vae/config.json estén presentes y que from_pretrained se ejecute correctamente.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- huggingface, python
- Área
- machine-learning
- Tipo de issue
- Error
- Dificultad
- 2/5
- Tiempo estimado
- 1-3 horas
- Estado de actividad
- Tranquilo
- Claridad
- Bien especificado
- Aptitud para principiantes
- 72/100