huggingface / huggingface/diffusers
Revisit modular diffusers weight-duplication workaround now that huggingface_hub has a shared blobs cache
- Ngôn ngữ chính
- Python
- Star
- 34.5k
- Fork
- 7.3k
- Merge trung bình
- 3 ngày 3 giờ
- Pull request đã merge (30 ngày)
- 91
Mô tả
`huggingface_hub v1.32.0` ships a **shared blobs cache**: Xet-backed files are now deduplicated locally *across repos* (best-effort, via symlinks, backward compatible with older versions and opt-out possible). See the [release notes](https://github.com/huggingface/huggingface_hub/releases/tag/v1.32.0) and [huggingface/huggingface_hub#4498](https://github.com/huggingface/huggingface_hub/pull/4498).
Concretely, downloading two repos that share components now only pays the download cost once — e.g. `Qwen/Qwen-Image` and `Qwen/Qwen-Image-Edit` share ~29% of their data (16.8G/57.7G), FLUX.1-dev / FLUX.1-schnell ~18% (10.3GB), `Kwai-Kolors/Kolors` / `Kolors-Inpainting` ~56% (>20GB).
This is relevant to modular diffusers repositories, which duplicate weights from the original repos. We currently have a workaround so that weights are fetched from the original repositories instead of the modular repo, precisely because cross-repo local dedup was missing. With the shared blobs cache in place, that workaround may no longer be necessary — modular repos would be a particularly striking use case for the new dedup (near-zero incremental download for duplicated weights).
Proposed follow-ups:
- Verify dedup actually kicks in for modular diffusers repos (Xet-enabled, identical blobs) and measure the gain.
- If confirmed, plan removal/simplification of the "download from original repo" workaround, bumping the `huggingface_hub` floor as needed and keeping a fallback for older versions.
---
Requested by Sayak Paul - [Slack thread](https://huggingface.slack.com/archives/C02V5EA0A95/p1789641097108929?thread_ts=1789641097.108929&cid=C02V5EA0A95) - [Agent trace](https://huggingface.co/buckets/huggingface/moon-bot-memory/tree/sessions/2026-09-17T13-09-28-443Z_e994893e-d33d-4980-be1f-5c024f5c84db.jsonl)
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.