huggingface / huggingface/diffusers

Revisit modular diffusers weight-duplication workaround now that huggingface_hub has a shared blobs cache

Abierto
#14,802 0 comentarios 0 reacciones 2 asignados Reclamado por @DN6 Ver en GitHub
Lenguaje dominante
Python
Estrellas
34.5k
Forks
7.3k
Merge medio
3 d 3 h
PR fusionados (30 d)
91

Descripción

`huggingface_hub v1.32.0` ships a **shared blobs cache**: Xet-backed files are now deduplicated locally *across repos* (best-effort, via symlinks, backward compatible with older versions and opt-out possible). See the [release notes](https://github.com/huggingface/huggingface_hub/releases/tag/v1.32.0) and [huggingface/huggingface_hub#4498](https://github.com/huggingface/huggingface_hub/pull/4498).

Concretely, downloading two repos that share components now only pays the download cost once — e.g. `Qwen/Qwen-Image` and `Qwen/Qwen-Image-Edit` share ~29% of their data (16.8G/57.7G), FLUX.1-dev / FLUX.1-schnell ~18% (10.3GB), `Kwai-Kolors/Kolors` / `Kolors-Inpainting` ~56% (>20GB).

This is relevant to modular diffusers repositories, which duplicate weights from the original repos. We currently have a workaround so that weights are fetched from the original repositories instead of the modular repo, precisely because cross-repo local dedup was missing. With the shared blobs cache in place, that workaround may no longer be necessary — modular repos would be a particularly striking use case for the new dedup (near-zero incremental download for duplicated weights).

Proposed follow-ups:
- Verify dedup actually kicks in for modular diffusers repos (Xet-enabled, identical blobs) and measure the gain.
- If confirmed, plan removal/simplification of the "download from original repo" workaround, bumping the `huggingface_hub` floor as needed and keeping a fallback for older versions.

---
Requested by Sayak Paul - [Slack thread](https://huggingface.slack.com/archives/C02V5EA0A95/p1789641097108929?thread_ts=1789641097.108929&cid=C02V5EA0A95) - [Agent trace](https://huggingface.co/buckets/huggingface/moon-bot-memory/tree/sessions/2026-09-17T13-09-28-443Z_e994893e-d33d-4980-be1f-5c024f5c84db.jsonl)

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.