invoke-ai / invoke-ai/InvokeAI

[enhancement]: prompt2prompt reweight operator

Open
#4,541 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
28.2k
Forks
3k
Avg merge
6d 5h
Merged PRs (30d)
19

Description

Two related things: 1) adding cross-attention controls other than `swap` (including `reweight`), 2) future of maintaining prompt2prompt implementation.

## Background

InvokeAI's implementation of [Prompt-to-Prompt Image Editing with Cross-Attention Control](https://prompt-to-prompt.github.io/) predates Invoke 2.3's adoption of `diffusers`. Often referred to as "prompt2prompt" in implementation discussions, it's documented as [Cross-Attention Control](https://invoke-ai.github.io/InvokeAI/features/PROMPTS/#cross-attention-control-prompt2prompt) in our user docs and the Word Swap control is accessed with the `(foo, bar).swap()` syntax in Invoke prompting.

The implementation has been one of the more complicated things to maintain, as it requires messing with those attention layers inside the U-Net instead of operating on its normal inputs and outputs.

## Recent Developments

diffusers recently accepted a PR for a prompt2prompt pipeline:
- https://github.com/huggingface/diffusers/pull/4563

I was hoping that would mean that Invoke would get to replace most or all of our implementation and use one maintained upstream. I'm also happy to see that it's a more complete implementation of prompt2prompt and includes the Refine control (adding new terms with blending) and Reweight (weighting based on attention, not by scaling the vectors from the text encoder).

Unfortunately diffusers has elected to add it to their `examples` as untested code instead of including it in the distribution, so we don't get out of having to maintain it after all.

# Next Steps

I think it probably makes sense to adopt the code from https://github.com/huggingface/diffusers/pull/4563 into Invoke's codebase, along with the [tests](https://github.com/huggingface/diffusers/blob/6e8afa4fdd255f8d6f606e84afa879ca5cf85fcd/tests/pipelines/prompt2prompt/test_prompt2prompt.py) that were written for it -- but TBH I haven't looked to see how much it differs from our current implementation.

Allowing the use of the other cross-attention control types would require supporting them in compel, or some other user interface.

We should also be thinking about how to keep feature parity between SD and SDXL. Presumably cross-attention controls may be applied to SDXL in a similar manner, but since SDXL's conditioning contains the output of _two_ text encoders, mapping between specific words in the text and the attention layers must be a little more complicated.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.