huggingface / huggingface/diffusers

DiffusionPipeline.download() on Windows silently skips all subfolder config.json files with huggingface_hub>=1.22.0

Open Beginner friendly
#14,142 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug pipelines
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

Describe the bug

On Windows, DiffusionPipeline.download() builds the allow patterns for per-component configs with os.path.join:
https://github.com/huggingface/diffusers/blob/b8905b9b0f01/src/diffusers/pipelines/pipeline_utils.py#L1740-L1741
which produces vae\config.json, while hub repo paths always use forward slashes (vae/config.json).

Until recently this was harmless: filter_repo_objects in huggingface_hub matched patterns with fnmatch.fnmatch, which on Windows runs os.path.normcase on both pattern and path and unifies the separators, so the backslash pattern still matched. huggingface_hub 1.22.0 switched it to fnmatch.fnmatchcase to make matching case-sensitive on all platforms (huggingface/huggingface_hub#4435), and fnmatchcase does no normalization. Since that release the backslash patterns match nothing:

>>> import os
>>> from fnmatch import fnmatchcase
>>> os.path.join("vae", "config.json")
'vae\\config.json'
>>> fnmatchcase("vae/config.json", os.path.join("vae", "config.json"))
False  # True on Linux/macOS, and True on Windows with fnmatch.fnmatch (hub <= 1.21.0)
Reproduction

Run on Windows with huggingface_hub>=1.22.0:

import os
from diffusers import DiffusionPipeline

folder = DiffusionPipeline.download("hf-internal-testing/tiny-stable-diffusion-torch", cache_dir="./fresh-cache")
for root, _, files in os.walk(folder):
    for f in files:
        print(os.path.relpath(os.path.join(root, f), folder).replace(os.sep, "/"))

Output: weights are fetched, but text_encoder/config.json, unet/config.json and vae/config.json are missing:

model_index.json
feature_extractor/preprocessor_config.json
scheduler/scheduler_config.json
text_encoder/pytorch_model.bin
tokenizer/merges.txt
tokenizer/special_tokens_map.json
tokenizer/tokenizer_config.json
tokenizer/vocab.json
unet/diffusion_pytorch_model.bin
vae/diffusion_pytorch_model.bin

DiffusionPipeline.from_pretrained("hf-internal-testing/tiny-stable-diffusion-torch", cache_dir="./fresh-cache") then fails, and keeps failing on every retry because the incomplete snapshot is considered fully cached. The same snippet on Linux, or on Windows with huggingface_hub<=1.21.0, also fetches the three config.json files.

Logs
`from_pretrained` failure with the reproduction above (transformers 5.x text encoder hits the missing config first and instantiates a default config, so the error is a shape mismatch):


  File "issue_repro.py", line 3, in <module>
    pipe = DiffusionPipeline.from_pretrained("hf-internal-testing/tiny-stable-diffusion-torch", cache_dir="./fresh-cache")
  File "...\site-packages\diffusers\pipelines\pipeline_utils.py", line 1057, in from_pretrained
    loaded_sub_model = load_sub_model(
  File "...\site-packages\diffusers\pipelines\pipeline_loading_utils.py", line 910, in load_sub_model
    loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
  File "...\site-packages\transformers\modeling_utils.py", line 4369, in from_pretrained
    loading_info = cls._finalize_model_loading(model, load_config, loading_info)
  File "...\site-packages\transformers\modeling_utils.py", line 4545, in _finalize_model_loading
    log_state_dict_report(
  File "...\site-packages\transformers\utils\loading_report.py", line 278, in log_state_dict_report
    raise RuntimeError(
RuntimeError: You set `ignore_mismatched_sizes` to `False`, thus raising an error. For details look at the above report!


When a diffusers-native component is the first one loaded from the cached folder (the original real-world case, a pipeline loaded with `transformer` and `text_encoder` passed in, so the VAE came from the snapshot):


  File "...\site-packages\diffusers\pipelines\pipeline_utils.py", line 1057, in from_pretrained
    loaded_sub_model = load_sub_model(
  File "...\site-packages\diffusers\pipelines\pipeline_loading_utils.py", line 910, in load_sub_model
    loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
  File "...\site-packages\diffusers\models\modeling_utils.py", line 1127, in from_pretrained
    config, unused_kwargs, commit_hash = cls.load_config(
  File "...\site-packages\diffusers\configuration_utils.py", line 418, in load_config
    raise EnvironmentError(
OSError: Error no file named config.json found in directory ...\models--vladmandic--Krea-2-Base-sdnq-hadamard-uint4\snapshots\c8ad74b656917d746124cb1aa3991c93a227c7dd\vae.
System Info
  • 🤗 Diffusers version: 0.39.0.dev0
  • Platform: Windows-11-10.0.26200-SP0
  • Running on Google Colab?: No
  • Python version: 3.13.11
  • PyTorch version (GPU?): 2.12.0+cu132 (True)
  • Flax version (CPU?/GPU?/TPU?): not installed (NA)
  • Jax version: not installed
  • JaxLib version: not installed
  • Huggingface_hub version: 1.22.0
  • Transformers version: 5.14.0.dev0
  • Accelerate version: 1.14.0
  • PEFT version: 0.19.1
  • Safetensors version: 0.8.0
  • xFormers version: not installed
  • Accelerator: NVIDIA RTX 2000 Ada Generation Laptop GPU, 8188 MiB
  • Using GPU in script?: No
  • Using distributed or parallel set-up in script?: No
Who can help?

@sayakpaul @DN6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/diffusers/pipelines/pipeline_utils.py around lines 1740-1741, where DiffusionPipeline.download() builds the component config allow patterns. Reproduce the download on Windows with huggingface_hub 1.22.0 using the tiny-stable-diffusion-torch example, then verify that text_encoder/config.json, unet/config.json, and vae/config.json are present and from_pretrained succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.