lllyasviel / lllyasviel/stable-diffusion-webui-forge

[Bug]: ControlNet not working with Inpaint in "Inpaint not masked" mode

Open
#482 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [ ] The issue has not been reported before recently
- [X] The issue has been reported before but has not been fixed yet

### What happened?

When using "Mask mode: Inpaint not masked", ControlNet does not work. This is inconsistent with the ControlNet implementation in Automatic1111/stable-diffusion-webui.

This issue is closely related to #76, which was fixed last month.

### Steps to reproduce the problem

1. Go to img2img/Inpaint
2. Type a simple prompt and drop a picture onto the canvas
3. Set Mask mode to "Inpaint not masked"
4. Set Denoising Strength to 1
5. Set ControlNet Unit 0 to Enable, Tile, tile_resample, control_v11f1e_sd15_tile, Control Weight 0.5
6. Generate an image. Notice that the generated image has no ControlNet Tile processing applied to it. The generated image looks exactly the same as when ControlNet is disabled. Also notice that the ControlNet input preview is a completely black image.

Two more observations:
1. When enabling "Upload independent control image" and using the same image, ControlNet works as expected.
2. Switching the Mask mode to "Inpaint masked" and drawing a mask that covers the entire image works as expected. Again, the expectation is that "Inpaint not masked" with no mask is analogous to "Inpaint masked" with a full mask, and should result in the same behavior. This is how ControlNet works in stable-diffusion-webui.

### What should have happened?

"Inpaint not masked" with no mask should behave the same as "Inpaint masked" with a full mask

### What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

### Sysinfo

[sysinfo-2024-03-04-11-12.json](https://github.com/lllyasviel/stable-diffusion-webui-forge/files/14480683/sysinfo-2024-03-04-11-12.json)

### Console logs

```Shell
venv "C:/Users/Admin/Desktop/SD/stable-diffusion-webui/venv\Scripts\Python.exe"
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: f0.0.17v1.8.0rc-latest-273-gb9705c58
Commit hash: b9705c58f66c6fd2c4a0168b26c5cf1fa6c0dde3
Launching Web UI with arguments: --xformers --pin-shared-memory --cuda-malloc --cuda-stream --ckpt-dir C:/Users/Admin/Desktop/SD/stable-diffusion-webui/models/Stable-diffusion --hypernetwork-dir C:/Users/Admin/Desktop/SD/stable-diffusion-webui/models/hypernetworks --embeddings-dir C:/Users/Admin/Desktop/SD/stable-diffusion-webui/embeddings --lora-dir C:/Users/Admin/Desktop/SD/stable-diffusion-webui/models/Lora
Using cudaMallocAsync backend.
Total VRAM 24564 MB, total RAM 32530 MB
WARNING:xformers:A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
xformers version: 0.0.23.post1
Set vram state to: NORMAL_VRAM
Always pin shared GPU memory
Device: cuda:0 NVIDIA GeForce RTX 3090 Ti : cudaMallocAsync
VAE dtype: torch.bfloat16
CUDA Stream Activated: True
2024-03-04 04:22:40.911261: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
WARNING:tensorflow:From C:\Users\Admin\Desktop\SD\stable-diffusion-webui\venv\lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

Using xformers cross attention
ControlNet preprocessor location: C:\Users\Admin\Desktop\SD\forge\models\ControlNetPreprocessor
Loading weights [3bb4fcd850] from C:\Users\Admin\Desktop\SD\stable-diffusion-webui\models\Stable-diffusion\sd_v1-5-pruned-emaonly.safetensors
2024-03-04 04:22:46,399 - ControlNet - INFO - ControlNet UI callback registered.
model_type EPS
UNet ADM Dimension 0
Running on local URL: http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 11.6s (prepare environment: 1.9s, import torch: 2.4s, import gradio: 0.6s, setup paths: 3.6s, other imports: 0.4s, load scripts: 1.4s, create ui: 1.0s, gradio launch: 0.1s).
Using xformers attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using xformers attention in VAE
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
left over keys: dict_keys(['alphas_cumprod', 'alphas_cumprod_prev', 'betas', 'log_one_minus_alphas_cumprod', 'model_ema.decay', 'model_ema.num_updates', 'posterior_log_variance_clipped', 'posterior_mean_coef1', 'posterior_mean_coef2', 'posterior_variance', 'sqrt_alphas_cumprod', 'sqrt_one_minus_alphas_cumprod', 'sqrt_recip_alphas_cumprod', 'sqrt_recipm1_alphas_cumprod'])
loaded straight to GPU
To load target model BaseModel
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) = 21591.593864440918
[Memory Management] Model Memory (MB) = 0.00762939453125
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 20567.586235046387
Moving model(s) has taken 0.01 seconds
To load target model SD1ClipModel
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) = 21590.358108520508
[Memory Management] Model Memory (MB) = 454.2076225280762
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 20112.15048599243
Moving model(s) has taken 0.05 seconds
Model loaded in 2.0s (load weights from disk: 0.1s, forge load real models: 1.3s, load VAE: 0.3s, calculate empty prompt: 0.2s).
2024-03-04 04:23:30,194 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
2024-03-04 04:23:30,270 - ControlNet - INFO - Using preprocessor: tile_resample
2024-03-04 04:23:30,270 - ControlNet - INFO - preprocessor resolution = 0.5
Warning torch.load doesn't support weights_only on this pytorch version, loading unsafely.
[] []
2024-03-04 04:23:30,859 - ControlNet - INFO - Current ControlNet ControlNetPatcher: C:\Users\Admin\Desktop\SD\forge\models\ControlNet\control_v11f1e_sd15_tile.pth
To load target model AutoencoderKL
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) = 21217.16962814331
[Memory Management] Model Memory (MB) = 159.55708122253418
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 20033.612546920776
Moving model(s) has taken 0.03 seconds
2024-03-04 04:23:31,467 - ControlNet - INFO - ControlNet Method tile_resample patched.
To load target model BaseModel
To load target model ControlNet
Begin to load 2 models
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) = 21084.882068634033
[Memory Management] Model Memory (MB) = 0.0
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 20060.882068634033
[Memory Management] Current Free GPU Memory (MB) = 21084.882068634033
[Memory Management] Model Memory (MB) = 689.0852355957031
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 19371.79683303833
Moving model(s) has taken 0.10 seconds
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:14<00:00, 1.28it/s]
```

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the listed img2img/Inpaint steps with ControlNet Unit 0 configured for Tile, then compare the ControlNet inpaint handling with Automatic1111/stable-diffusion-webui. Done means “Inpaint not masked” with no mask produces the same ControlNet behavior as “Inpaint masked” with a full-image mask, including a non-black input preview.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.