huggingface / huggingface/diffusers

Revisit modular diffusers weight-duplication workaround now that huggingface_hub has a shared blobs cache

Ouverte
#14,802 0 commentaires 0 réactions 2 personnes assignées Réclamée par @DN6 Voir sur GitHub
Langage dominant
Python
Étoiles
34.5k
Forks
7.3k
Merge moyen
3 j 3 h
PR mergées (30 j)
91

Description

`huggingface_hub v1.32.0` ships a **shared blobs cache**: Xet-backed files are now deduplicated locally *across repos* (best-effort, via symlinks, backward compatible with older versions and opt-out possible). See the [release notes](https://github.com/huggingface/huggingface_hub/releases/tag/v1.32.0) and [huggingface/huggingface_hub#4498](https://github.com/huggingface/huggingface_hub/pull/4498).

Concretely, downloading two repos that share components now only pays the download cost once — e.g. `Qwen/Qwen-Image` and `Qwen/Qwen-Image-Edit` share ~29% of their data (16.8G/57.7G), FLUX.1-dev / FLUX.1-schnell ~18% (10.3GB), `Kwai-Kolors/Kolors` / `Kolors-Inpainting` ~56% (>20GB).

This is relevant to modular diffusers repositories, which duplicate weights from the original repos. We currently have a workaround so that weights are fetched from the original repositories instead of the modular repo, precisely because cross-repo local dedup was missing. With the shared blobs cache in place, that workaround may no longer be necessary — modular repos would be a particularly striking use case for the new dedup (near-zero incremental download for duplicated weights).

Proposed follow-ups:
- Verify dedup actually kicks in for modular diffusers repos (Xet-enabled, identical blobs) and measure the gain.
- If confirmed, plan removal/simplification of the "download from original repo" workaround, bumping the `huggingface_hub` floor as needed and keeping a fallback for older versions.

---
Requested by Sayak Paul - [Slack thread](https://huggingface.slack.com/archives/C02V5EA0A95/p1789641097108929?thread_ts=1789641097.108929&cid=C02V5EA0A95) - [Agent trace](https://huggingface.co/buckets/huggingface/moon-bot-memory/tree/sessions/2026-09-17T13-09-28-443Z_e994893e-d33d-4980-be1f-5c024f5c84db.jsonl)

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.