Comfy-Org / Comfy-Org/ComfyUI

Auto-generated mask for conditioning based on prompt attention

Open
#5,473 1 comment 0 reactions 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Feature Idea

There currently exists a `Conditioning (Set Mask)` node, but masks need to be created manually ahead of time for this. I propose the ability to feed in auto-generated masks based on a prompt input, which would allow automatically creating images like this ([source](https://github.com/hako-mikan/sd-webui-regional-prompter?tab=readme-ov-file#region-specification-by-prompt-experimental)). In the below example, `shirt` is the prompt attention/conditioning target, and `apple printed` is the conditioning applied to that area.
![psample6](https://github.com/user-attachments/assets/56cd843c-014b-47a1-a863-2f3074bda53a)
Because the composition of an image can vastly change seed to seed, it is much more convenient if the mask is generated on-the-fly.

### Existing Solutions

[sd-webui-regional-prompter](https://github.com/hako-mikan/sd-webui-regional-prompter?tab=readme-ov-file#region-specification-by-prompt-experimental) is the only existing solution I'm aware of. They refer to this as "region specification by prompt". It supports both an attention and a latent mode (latent internally having a larger size, making masks like this more accurate).

Example of Attention vs. Latent

![image](https://github.com/user-attachments/assets/d3271d22-1498-4470-8b91-a2188bef3a65)


---

There also exists [ComfyUI-CLIPSeg](https://github.com/biegert/ComfyUI-CLIPSeg), but unlike sd-webui-regional-prompter, this relies on using an existing image rather than generating the mask at the time of inference.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.