lllyasviel / lllyasviel/stable-diffusion-webui-forge

[Feature Request]: Add support for PuLID

Open
#722 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
13k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What would your feature do ?

see this discussion: https://github.com/Mikubill/sd-webui-controlnet/discussions/2841

> [PuLID](https://github.com/ToTheBeginning/PuLID) is an ip-adapter alike method to restore facial identity. It uses both insightface embedding and CLIP embedding similar to what ip-adapter faceid plus model does. However, there is an extra process of masking out the face from background environment using facexlib before passing image to CLIP. PuLID also uses Eva CLIP instead of normal CLIP. In the attn overrides, PuLID also does something more than IPAdapter, as it zero pads the tensor and adds to ortho of hidden states. If you are interested you can read their paper on how this is handled.

Please add support for it.

Maybe ip-adapter-auto too.

### Proposed workflow

merge it from original ContrctNet

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked GitHub discussion and the PuLID project and paper to understand the requested integration, then compare it with the original ControlNet implementation mentioned in the proposed workflow. Done means implementing and validating PuLID support, including its identity embeddings, face masking, Eva CLIP processing, and attention behavior; no project files or tests are named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, computer-vision, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.