invoke-ai / invoke-ai/InvokeAI
[enhancement]: prompt2prompt reweight operator
- Dominant language
- Python
- Stars
- 28.2k
- Forks
- 3k
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 19
Description
Two related things: 1) adding cross-attention controls other than `swap` (including `reweight`), 2) future of maintaining prompt2prompt implementation.
## Background
InvokeAI's implementation of [Prompt-to-Prompt Image Editing with Cross-Attention Control](https://prompt-to-prompt.github.io/) predates Invoke 2.3's adoption of `diffusers`. Often referred to as "prompt2prompt" in implementation discussions, it's documented as [Cross-Attention Control](https://invoke-ai.github.io/InvokeAI/features/PROMPTS/#cross-attention-control-prompt2prompt) in our user docs and the Word Swap control is accessed with the `(foo, bar).swap()` syntax in Invoke prompting.
The implementation has been one of the more complicated things to maintain, as it requires messing with those attention layers inside the U-Net instead of operating on its normal inputs and outputs.
## Recent Developments
diffusers recently accepted a PR for a prompt2prompt pipeline:
- https://github.com/huggingface/diffusers/pull/4563
I was hoping that would mean that Invoke would get to replace most or all of our implementation and use one maintained upstream. I'm also happy to see that it's a more complete implementation of prompt2prompt and includes the Refine control (adding new terms with blending) and Reweight (weighting based on attention, not by scaling the vectors from the text encoder).
Unfortunately diffusers has elected to add it to their `examples` as untested code instead of including it in the distribution, so we don't get out of having to maintain it after all.
# Next Steps
I think it probably makes sense to adopt the code from https://github.com/huggingface/diffusers/pull/4563 into Invoke's codebase, along with the [tests](https://github.com/huggingface/diffusers/blob/6e8afa4fdd255f8d6f606e84afa879ca5cf85fcd/tests/pipelines/prompt2prompt/test_prompt2prompt.py) that were written for it -- but TBH I haven't looked to see how much it differs from our current implementation.
Allowing the use of the other cross-attention control types would require supporting them in compel, or some other user interface.
We should also be thinking about how to keep feature parity between SD and SDXL. Presumably cross-attention controls may be applied to SDXL in a similar manner, but since SDXL's conditioning contains the output of _two_ text encoders, mapping between specific words in the text and the attention layers must be a little more complicated.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.