huggingface / huggingface/diffusers

Standardization of additional token identifiers across pipelines

Aperta
#11,334 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
stale
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

`FluxPipeline` has utilities that give us `img_ids` and `txt_ids`:

https://github.com/huggingface/diffusers/blob/ce1063acfa0cbc2168a7e9dddd4282ab8013b810/src/diffusers/pipelines/flux/pipeline_flux.py#L514

https://github.com/huggingface/diffusers/blob/ce1063acfa0cbc2168a7e9dddd4282ab8013b810/src/diffusers/pipelines/flux/pipeline_flux.py#L385

As such these are not created inside the `transformer` class.

Whereas in `HiDream`, we have something different.

`text_ids` are created inside the `transformer` class:
https://github.com/huggingface/diffusers/blob/ce1063acfa0cbc2168a7e9dddd4282ab8013b810/src/diffusers/models/transformers/transformer_hidream_image.py#L796

`img_ids` are overwritten:
https://github.com/huggingface/diffusers/blob/ce1063acfa0cbc2168a7e9dddd4282ab8013b810/src/diffusers/models/transformers/transformer_hidream_image.py#L771C13-L771C20 (probably intentional because it's conditioned)

Then the entire computation

https://github.com/huggingface/diffusers/blob/ce1063acfa0cbc2168a7e9dddd4282ab8013b810/src/diffusers/pipelines/hidream_image/pipeline_hidream_image.py#L726-L744

happens inside the pipeline `__call__()`. Maybe this could take place inside a method similar to the `FluxPipeline`?

In general, these could be standardized a bit.

Cc: @yiyixuxu @a-r-r-o-w

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Compare the linked implementations in src/diffusers/pipelines/flux/pipeline_flux.py, src/diffusers/models/transformers/transformer_hidream_image.py, and src/diffusers/pipelines/hidream_image/pipeline_hidream_image.py. First determine the intended ownership and shape of img_ids and text_ids across both pipelines; done means the standardization approach is agreed and the affected pipeline behavior remains consistent.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning
Tipo di issue
Refactoring
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
32/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.