lllyasviel / lllyasviel/stable-diffusion-webui-forge
[Feature Request]: Add support for PuLID
- Dominant language
- Python
- Stars
- 13k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues and checked the recent builds/commits
### What would your feature do ?
see this discussion: https://github.com/Mikubill/sd-webui-controlnet/discussions/2841
> [PuLID](https://github.com/ToTheBeginning/PuLID) is an ip-adapter alike method to restore facial identity. It uses both insightface embedding and CLIP embedding similar to what ip-adapter faceid plus model does. However, there is an extra process of masking out the face from background environment using facexlib before passing image to CLIP. PuLID also uses Eva CLIP instead of normal CLIP. In the attn overrides, PuLID also does something more than IPAdapter, as it zero pads the tensor and adds to ortho of hidden states. If you are interested you can read their paper on how this is handled.
Please add support for it.
Maybe ip-adapter-auto too.
### Proposed workflow
merge it from original ContrctNet
### Additional information
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked GitHub discussion and the PuLID project and paper to understand the requested integration, then compare it with the original ControlNet implementation mentioned in the proposed workflow. Done means implementing and validating PuLID support, including its identity embeddings, face masking, Eva CLIP processing, and attention behavior; no project files or tests are named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100