invoke-ai / invoke-ai/InvokeAI

[enhancement]: Any plan to support ControlNet's promptless Inpaint? (or IP-Adapter's)

Open
#5,340 0 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
28.2k
Forks
3k
Avg merge
6d 5h
Merged PRs (30d)
19

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Contact Details

_No response_

### What should this feature add?

It was 6 months ago, controlnet published a new model call "inpaint", with that you can do promptless inpaintings with results comparable to Adobe's Firefly ([link](https://github.com/Mikubill/sd-webui-controlnet/discussions/1464)). The advantage of controlnet inpainting is not only promptless, but also the ability to work with any model and lora you desire, instead of just inpainting models.

InvokeAI still lacks such a functionality. There is an inpaint controlnet mode, but the required preprocessors are missing. Specifically, the "inpaint-global-harmonious" and "inpaint-only".
Without those preprocessors, most of the time we can only fallback to the old sd1.5 inpainting, which doesn't support promptless inpainting at all.

Here are some tests:
infill method is set to "Lama", all results are non-cherrypicked.

Original Image and the inpaint Mask (the right part is not shown here),
As you can see, this is the kind of image with stylish cartoon touches. A bare model without Lora or prompt engineering is expected to fail to some extents.
![invokeai-mask](https://github.com/invoke-ai/InvokeAI/assets/13725477/c06c6334-a549-4b16-a222-28177083753f)

1. InvokeAI, revAnimated with Lora, ControlNet "Inpaint", preprocessor "None" (None is chosen automatically when you select inpaint), prompt ""
![invokeai-control-noprompt-rev](https://github.com/invoke-ai/InvokeAI/assets/13725477/23a31ab6-a2ec-4626-96c2-b982ed521222)
Barely any changes.

Now, decrease the control weight to 0.7:
![invokeai-control-noprompt-rev-weight0 7](https://github.com/invoke-ai/InvokeAI/assets/13725477/dff3f37c-1bbe-41f2-afca-67fa53bf446b)

Still not an "inpainting" work

2. InvokeAI, sd1.5-inpaint-model no Lora, no ControlNet, prompt ""
![invokeai-sdinpaint-1](https://github.com/invoke-ai/InvokeAI/assets/13725477/ded28562-ed31-4ff9-be39-77e573c44927)
![invokeai-sdinpaint-2](https://github.com/invoke-ai/InvokeAI/assets/13725477/9edfce6c-9bd2-4b09-8d36-d2b121f18852)

3. InvokeAI, sd1.5-inpaint-model no Lora, no ControlNet, prompt "dining table, chairs"
![invokeai-sdinpaint-prompted-0](https://github.com/invoke-ai/InvokeAI/assets/13725477/738de998-d0ea-4121-bac1-acc83cbff591)
![invokeai-sdinpaint-prompted-1](https://github.com/invoke-ai/InvokeAI/assets/13725477/92c841cd-47b0-461c-a54c-29cfebcaa470)

It finally gives some meaningful results, but clearly the barebone inpainting mode is struggling with the art style.

4. A111, revAnimated model, ControlNet "Inpaint", Preprocessor "Inpaint+Lama", Prompt ""
![a111-control-noprompt-rev](https://github.com/invoke-ai/InvokeAI/assets/13725477/9c131b39-f84c-497f-b7dc-f3d3a3cf2f2c)
![a111-control-noprompt-rev-2](https://github.com/invoke-ai/InvokeAI/assets/13725477/a0f3053a-fafd-4429-8d7f-43add6800f2b)

The results are immediately better, the style matches too.

5. Finally, A111, revAnimated model, ControlNet "Inpaint", Preprocessor "Inpaint+Lama", Prompt "dining table, chairs"
![a111-control-withprompt-rev-1](https://github.com/invoke-ai/InvokeAI/assets/13725477/08c8f124-0473-445c-abb1-fc1cf4f844eb)
![a111-control-withprompt-rev-2](https://github.com/invoke-ai/InvokeAI/assets/13725477/841aa722-c4c2-46ba-94bd-1a5f08cca3b8)

I wouldn't call this perfect, but they are a clear winner against InvokeAI's old sd1.5 inpaint model.

Some users reported that using IP-Adapter can also achieve similar [great results](https://www.reddit.com/r/StableDiffusion/comments/15yyvzy/promptless_inpaint_generative_fill_with_ipadapter/), but the node only exists for ComfyUI. InvokeAI's IP-Adapter UI panel lacks the essential auto preprocessing tool (auto take the base image and the mask combined as input), you have to do it manually every time the mask changes.

So, my question is: is there any plan to support the ControlNet's or IP-Adapter's Inpaint method? InvokeAI's Unified Canvas is inarguably the best UI around, actually that's the reason I like InvokeAI so much, but sadly its core processing ability lacks some punches.

### Alternatives

_No response_

### Additional Content

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.