huggingface / huggingface/diffusers

Potential incorrect reshaping in 2D positional embedding

Offen
#11,309 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug stale
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

### Describe the bug

Hi there,

I have concerns with this line of code (https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/embeddings.py#L282).

Specifically, `grid_size` is the tuple consisting of the height `H` and width `W` of the image. `grid` computed in L280 should have the shape `2*H*W`, and L282 reshapes it into `2*1*W*H`. The dimensions `W*H` will be later flattened to match the dimensions of the latent.

However, if you continue to `PatchEmbed` (https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/embeddings.py#L549), you will notice that the latent with shape `BCHW` is flattened into `B(H*W)C`, this flattening operation does not seem to match with `grid` in L282. I think this reordering will mess up the ordering of dimensions when being flattened in case `H` and `W` are not equal.

### Reproduction

This potential bug is conceptual and no need for reproduction.

### Logs

```shell

```

### System Info

Current `diffusers` implementation.

### Who can help?

@yiyixuxu @sayakpaul

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start in src/diffusers/models/embeddings.py at the grid construction around line 280 and the PatchEmbed implementation around line 549. Compare the positional grid ordering with the BCHW-to-B(H*W)C flattening for unequal height and width, then confirm whether the ordering matches and add a regression test or correction if it does not.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
machine-learning
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.