lllyasviel / lllyasviel/stable-diffusion-webui-forge
[Bug]: ControlNet inpaint incorrectly cropped mask
- Dominant language
- Python
- Stars
- 13k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
### Checklist
- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet
### What happened?
When using ControlNet inpainting with resize mode set to 'crop and resize' the black and white mask image passed to ControlNet is cropped incorrectly.
### Example:
Original image:
Inpaint settings, resolution is 1024x1024:

Cropped outputs stacked on top, mask is clearly misaligned and cropped:

### Steps to reproduce the problem
1. Go to img2img inpaint
2. Input the example image above
3. Paint mask over strings and yellow balloon
4. Set inpaint settings, resolution 1024*1024:

5. Set ControlNet settings:

6. Generate
### What should have happened?
The yellow balloon should have been painted out
Example when settings inpaint area to 'Whole picture':

### What browsers do you use to access the UI ?
Mozilla Firefox
### Sysinfo
[sysinfo-2024-02-21-20-13.json](https://github.com/lllyasviel/stable-diffusion-webui-forge/files/14364558/sysinfo-2024-02-21-20-13.json)
### Console logs
```Shell
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: f0.0.14v1.8.0rc-latest-184-g43c9e3b5
Commit hash: 43c9e3b5ce1642073c7a9684e36b45489eeb4a49
Launching Web UI with arguments:
Total VRAM 12287 MB, total RAM 32717 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3060 : native
VAE dtype: torch.bfloat16
Using pytorch cross attention
ControlNet preprocessor location: G:\AI\clean forge\webui\models\ControlNetPreprocessor
Loading weights [821aa5537f] from G:\AI\clean forge\webui\models\Stable-diffusion\autismmixSDXL_autismmixPony.safetensors
2024-02-21 20:30:57,792 - ControlNet - INFO - ControlNet UI callback registered.
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
model_type EPS
UNet ADM Dimension 2816
Startup time: 30.1s (prepare environment: 8.0s, import torch: 7.5s, import gradio: 2.8s, setup paths: 2.7s, initialize shared: 0.2s, other imports: 1.8s, load scripts: 3.5s, create ui: 1.1s, gradio launch: 2.7s).
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids', 'cond_stage_model.clip_l.logit_scale'}
Loading VAE weights specified in settings: G:\AI\clean forge\webui\models\VAE\sdxl_vae.safetensors
To load target model SDXLClipModel
Begin to load 1 model
Moving model(s) has taken 0.62 seconds
Model loaded in 11.8s (load weights from disk: 0.8s, forge solving config: 0.2s, forge instantiate config: 2.3s, forge load real models: 7.0s, load VAE: 0.4s, calculate empty prompt: 1.0s).
2024-02-21 20:32:41,665 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
2024-02-21 20:32:41,818 - ControlNet - INFO - Crop input image based on A1111 mask.
2024-02-21 20:32:42,022 - ControlNet - INFO - Crop input image based on A1111 mask.
2024-02-21 20:32:42,308 - ControlNet - INFO - Using preprocessor: inpaint_only
2024-02-21 20:32:42,309 - ControlNet - INFO - preprocessor resolution = 0.5
Warning torch.load doesn't support weights_only on this pytorch version, loading unsafely.
2024-02-21 20:32:43,469 - ControlNet - INFO - Current ControlNet FooocusInpaintPatcher: G:\AI\clean forge\webui\models\ControlNet\inpaint_v26.fooocus.patch
To load target model AutoencoderKL
Begin to load 1 model
Moving model(s) has taken 0.33 seconds
[Fooocus Patch Loader] 960 keys loaded, 0 remaining keys not found in model.
2024-02-21 20:32:45,702 - ControlNet - INFO - ControlNet Method inpaint_only patched.
To load target model SDXL
Begin to load 1 model
Moving model(s) has taken 2.53 seconds
27%|█████████████████████████████ | 8/30 [00:05<00:15, 1.41it/s]To load target model SDXL█████████████████████▊ | 8/30 [00:04<00:14, 1.47it/s]
Begin to load 1 model
unload clone 0
Moving model(s) has taken 2.00 seconds
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:23<00:00, 1.30it/s]
Moving model(s) skipped. Freeing memory has taken 0.53 seconds███████████████████████████████████████████████████| 30/30 [00:22<00:00, 1.44it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:24<00:00, 1.22it/s]
```
### Additional information
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue from the img2img inpaint flow using ControlNet with resize mode set to “crop and resize” and a 1024×1024 resolution. Compare the masked output with the “Whole picture” inpaint-area behavior, then trace the ControlNet crop and `inpaint_only` preprocessing path. Done means the mask remains aligned and the yellow balloon is painted out as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100