huggingface / huggingface/diffusers

Revisit modular diffusers weight-duplication workaround now that huggingface_hub has a shared blobs cache

Đang mở
#14,802 0 bình luận 0 reaction 2 người được giao Được @DN6 nhận Xem trên GitHub
Ngôn ngữ chính
Python
Star
34.5k
Fork
7.3k
Merge trung bình
3 ngày 3 giờ
Pull request đã merge (30 ngày)
91

Mô tả

`huggingface_hub v1.32.0` ships a **shared blobs cache**: Xet-backed files are now deduplicated locally *across repos* (best-effort, via symlinks, backward compatible with older versions and opt-out possible). See the [release notes](https://github.com/huggingface/huggingface_hub/releases/tag/v1.32.0) and [huggingface/huggingface_hub#4498](https://github.com/huggingface/huggingface_hub/pull/4498).

Concretely, downloading two repos that share components now only pays the download cost once — e.g. `Qwen/Qwen-Image` and `Qwen/Qwen-Image-Edit` share ~29% of their data (16.8G/57.7G), FLUX.1-dev / FLUX.1-schnell ~18% (10.3GB), `Kwai-Kolors/Kolors` / `Kolors-Inpainting` ~56% (>20GB).

This is relevant to modular diffusers repositories, which duplicate weights from the original repos. We currently have a workaround so that weights are fetched from the original repositories instead of the modular repo, precisely because cross-repo local dedup was missing. With the shared blobs cache in place, that workaround may no longer be necessary — modular repos would be a particularly striking use case for the new dedup (near-zero incremental download for duplicated weights).

Proposed follow-ups:
- Verify dedup actually kicks in for modular diffusers repos (Xet-enabled, identical blobs) and measure the gain.
- If confirmed, plan removal/simplification of the "download from original repo" workaround, bumping the `huggingface_hub` floor as needed and keeping a fallback for older versions.

---
Requested by Sayak Paul - [Slack thread](https://huggingface.slack.com/archives/C02V5EA0A95/p1789641097108929?thread_ts=1789641097.108929&cid=C02V5EA0A95) - [Agent trace](https://huggingface.co/buckets/huggingface/moon-bot-memory/tree/sessions/2026-09-17T13-09-28-443Z_e994893e-d33d-4980-be1f-5c024f5c84db.jsonl)

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.