Auto-generated mask for conditioning based on prompt attention
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Feature Idea
There currently exists a `Conditioning (Set Mask)` node, but masks need to be created manually ahead of time for this. I propose the ability to feed in auto-generated masks based on a prompt input, which would allow automatically creating images like this ([source](https://github.com/hako-mikan/sd-webui-regional-prompter?tab=readme-ov-file#region-specification-by-prompt-experimental)). In the below example, `shirt` is the prompt attention/conditioning target, and `apple printed` is the conditioning applied to that area.

Because the composition of an image can vastly change seed to seed, it is much more convenient if the mask is generated on-the-fly.
### Existing Solutions
[sd-webui-regional-prompter](https://github.com/hako-mikan/sd-webui-regional-prompter?tab=readme-ov-file#region-specification-by-prompt-experimental) is the only existing solution I'm aware of. They refer to this as "region specification by prompt". It supports both an attention and a latent mode (latent internally having a larger size, making masks like this more accurate).
Example of Attention vs. Latent

---
There also exists [ComfyUI-CLIPSeg](https://github.com/biegert/ComfyUI-CLIPSeg), but unlike sd-webui-regional-prompter, this relies on using an existing image rather than generating the mask at the time of inference.
Contributor guide
Assessment
This issue has not been assessed yet.