huggingface / huggingface/diffusers

Is that a bug of prompt2prompt pipeline with replace word pormpt?

Open
#12,881 1 comment 0 reactions 0 assignees View on GitHub
bug stale
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Describe the bug

It performance the same when return different cross attention map, is implement error or just the problem with prompt2prompt.

### Reproduction

Use stable-diffusion-2-1:
`images = pipe(["A turtle playing with a ball", "A monkey playing with a ball"],
generator=torch.Generator("cuda").manual_seed(34),
cross_attention_kwargs={
"edit_type": "replace",
"local_blend_words": ["turtle", "monkey"],
"n_cross_replace": 0.4,
"n_self_replace": 0.4
}).images`

It performance the same when return different cross attention map:
`class AttentionReplace(AttentionControlEdit):
def replace_cross_attention(self, attn_base, att_replace):
return attn_base.unsqueeze(0).expand(att_replace.shape[0], *attn_base.shape)
return torch.einsum("hpw,bwn->bhpn", attn_base, self.mapper)`

### Logs

```shell

```

### System Info

Diffusers=0.30.0

### Who can help?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the prompt2prompt pipeline reproduction in the issue and inspect AttentionReplace.replace_cross_attention, comparing the returned cross-attention maps for the turtle and monkey prompts. Reproduce the example with Diffusers 0.30.0 and determine whether identical output is an implementation error or expected prompt2prompt behavior; document the cause and add a regression check if the repository’s existing tests provide a suitable entry point.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.