lllyasviel / lllyasviel/stable-diffusion-webui-forge

[Bug]: Controlnet intergrated broken after update

Open

#467 opened on Mar 2, 2024 13 comments 5 reactions 0 assignees View on GitHub

Dominant language: Python
Stars: 13k
Forks: 1.7k
PR merge metrics: No merged PRs in 30d

Description

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [ ] The issue exists in the current version of the webui
- [ ] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

All controlnet are not working after update. This is the image generate with controlnet depth:

![image](https://github.com/lllyasviel/stable-diffusion-webui-forge/assets/77379748/9bdcab6f-cac2-421b-bb8f-5015af6098ae)

### Steps to reproduce the problem

1. Fresh install Forge
2. Begin a simple prompt
3. Use Realistic Vision which Forge automatic download
4. Using controlnet
5. Get error log and image does not follow controlnet guidance

### What should have happened?

After update some codes link to controlnet are broken or missing maybe

### What browsers do you use to access the UI ?

Google Chrome

### Sysinfo

[sysinfo-2024-03-02-16-55.json](https://github.com/lllyasviel/stable-diffusion-webui-forge/files/14470132/sysinfo-2024-03-02-16-55.json)

### Console logs

```Shell
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: f0.0.9-latest-51-g5bea443d
Commit hash: 5bea443d94f3a85f819cb8541c1bba0aac208d83
Installing clip
Installing open_clip
Installing requirements
Installing forge_legacy_preprocessor requirement: fvcore
Installing forge_legacy_preprocessor requirement: mediapipe
Installing forge_legacy_preprocessor requirement: svglib
Launching Web UI with arguments:
Total VRAM 24575 MB, total RAM 32672 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3090 : native
VAE dtype: torch.bfloat16
Using pytorch cross attention
Downloading: "https://huggingface.co/lllyasviel/fav_models/resolve/main/fav/realisticVisionV51_v51VAE.safetensors" to C:\AI\Forgetest\webui\models\Stable-diffusion\realisticVisionV51_v51VAE.safetensors

100%|█████████████████████████████████████████████████████████████████████████████| 1.99G/1.99G [01:50<00:00, 19.4MB/s]
ControlNet preprocessor location: C:\AI\Forgetest\webui\models\ControlNetPreprocessor
Calculating sha256 for C:\AI\Forgetest\webui\models\Stable-diffusion\realisticVisionV51_v51VAE.safetensors: 2024-03-02 23:48:09,824 - ControlNet - INFO - ControlNet UI callback registered.
Running on local URL: http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 157.8s (prepare environment: 38.6s, import torch: 3.4s, import gradio: 0.9s, setup paths: 0.7s, initialize shared: 0.2s, other imports: 0.7s, list SD models: 110.7s, load scripts: 1.8s, create ui: 0.4s, gradio launch: 0.4s).
15012c538f503ce2ebfc2c8547b268c75ccdaff7a281db55399940ff1d70e21d
Loading weights [15012c538f] from C:\AI\Forgetest\webui\models\Stable-diffusion\realisticVisionV51_v51VAE.safetensors
model_type EPS
UNet ADM Dimension 0
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
loaded straight to GPU
To load target model BaseModel
Begin to load 1 model
To load target model SD1ClipModel
Begin to load 1 model
Model loaded in 4.5s (calculate hash: 2.5s, forge load real models: 1.8s, calculate empty prompt: 0.2s).
2024-03-02 23:48:45,581 - ControlNet - INFO - Preview Resolution = 512
Downloading: "https://huggingface.co/lllyasviel/ControlNet/resolve/main/annotator/ckpts/dpt_hybrid-midas-501f0c75.pt" to C:\AI\Forgetest\webui\models\ControlNetPreprocessor\midas\dpt_hybrid-midas-501f0c75.pt

100%|███████████████████████████████████████████████████████████████████████████████| 470M/470M [00:25<00:00, 19.6MB/s]
Automatic Memory Management: 659 Modules in 0.10 seconds.
2024-03-02 23:49:19,441 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
2024-03-02 23:49:19,443 - ControlNet - INFO - Using preprocessor: depth_midas
2024-03-02 23:49:19,444 - ControlNet - INFO - preprocessor resolution = 512
Automatic Memory Management: 11 Modules in 0.15 seconds.
*** Error running process: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 798, in process
script.process(p, *script_args)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 450, in process
self.process_unit_after_click_generate(p, unit, params, *args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 333, in process_unit_after_click_generate
assert unit.model != 'None', 'You have not selected any control model!'
AssertionError: You have not selected any control model!

---
*** Error running process_before_every_sampling: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 830, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 457, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
KeyError: 0

---
100%|██████████████████████████████████████████████████████████████████████████████████| 35/35 [00:02<00:00, 12.16it/s]
To load target model AutoencoderKL██▏ | 35/150 [00:02<00:08, 13.21it/s]
Begin to load 1 model
Upscale script freed memory successfully.
Downloading: "https://github.com/cszn/KAIR/releases/download/v1.0/ESRGAN.pth" to C:\AI\Forgetest\webui\models\ESRGAN\ESRGAN_4x.pth

100%|█████████████████████████████████████████████████████████████████████████████| 63.8M/63.8M [00:10<00:00, 6.46MB/s]
tiled upscale: 100%|███████████████████████████████████████████████████████████████████| 15/15 [00:01<00:00, 8.23it/s]
*** Error running process_before_every_sampling: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 830, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 457, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
KeyError: 0

---
100%|██████████████████████████████████████████████████████████████████████████████████| 15/15 [00:07<00:00, 1.91it/s]
*** Error running postprocess_batch_list: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 854, in postprocess_batch_list
script.postprocess_batch_list(p, pp, *script_args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 463, in postprocess_batch_list
self.process_unit_after_every_sampling(p, unit, self.current_params[i], pp, *args, **kwargs)
KeyError: 0

---
100%|██████████████████████████████████████████████████████████████████████████████████| 35/35 [00:02<00:00, 13.17it/s]
Upscale script freed memory successfully.████████████████▊ | 85/150 [00:29<00:05, 12.98it/s]
tiled upscale: 100%|███████████████████████████████████████████████████████████████████| 15/15 [00:01<00:00, 9.08it/s]
*** Error running process_before_every_sampling: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 830, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 457, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
KeyError: 0

---
100%|██████████████████████████████████████████████████████████████████████████████████| 35/35 [00:02<00:00, 13.15it/s]
Upscale script freed memory successfully.█████████████████████████████████████▌ | 135/150 [00:42<00:01, 12.92it/s]
tiled upscale: 100%|███████████████████████████████████████████████████████████████████| 15/15 [00:01<00:00, 9.05it/s]
*** Error running process_before_every_sampling: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 830, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 457, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
KeyError: 0

---
100%|██████████████████████████████████████████████████████████████████████████████████| 15/15 [00:07<00:00, 1.88it/s]
*** Error running postprocess_batch_list: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 854, in postprocess_batch_list
script.postprocess_batch_list(p, pp, *script_args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 463, in postprocess_batch_list
self.process_unit_after_every_sampling(p, unit, self.current_params[i], pp, *args, **kwargs)
KeyError: 0

---
Total progress: 100%|████████████████████████████████████████████████████████████████| 150/150 [00:55<00:00, 2.69it/s]
2024-03-02 23:50:24,545 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE| 150/150 [00:55<00:00, 1.86it/s]
2024-03-02 23:50:24,547 - ControlNet - INFO - Using preprocessor: depth_midas
2024-03-02 23:50:24,548 - ControlNet - INFO - preprocessor resolution = 512
Automatic Memory Management: 11 Modules in 0.13 seconds.
*** Error running process: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 798, in process
script.process(p, *script_args)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 450, in process
self.process_unit_after_click_generate(p, unit, params, *args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 333, in process_unit_after_click_generate
assert unit.model != 'None', 'You have not selected any control model!'
AssertionError: You have not selected any control model!

---
100%|██████████████████████████████████████████████████████████████████████████████████| 35/35 [00:02<00:00, 12.77it/s]
Upscale script freed memory successfully. | 35/150 [00:02<00:08, 13.15it/s]
tiled upscale: 100%|███████████████████████████████████████████████████████████████████| 15/15 [00:01<00:00, 8.93it/s]
*** Error running process_before_every_sampling: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 830, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 457, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
KeyError: 0

---
100%|██████████████████████████████████████████████████████████████████████████████████| 35/35 [00:02<00:00, 13.15it/s]
Upscale script freed memory successfully.████████████████▊ | 85/150 [00:16<00:05, 12.95it/s]
tiled upscale: 100%|███████████████████████████████████████████████████████████████████| 15/15 [00:01<00:00, 9.03it/s]
*** Error running process_before_every_sampling: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 830, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 457, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
KeyError: 0

---
100%|██████████████████████████████████████████████████████████████████████████████████| 35/35 [00:02<00:00, 13.10it/s]
Upscale script freed memory successfully.█████████████████████████████████████▌ | 135/150 [00:30<00:01, 12.94it/s]
tiled upscale: 100%|███████████████████████████████████████████████████████████████████| 15/15 [00:01<00:00, 9.09it/s]
*** Error running process_before_every_sampling: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 830, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 457, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
KeyError: 0

---
100%|██████████████████████████████████████████████████████████████████████████████████| 15/15 [00:08<00:00, 1.86it/s]
*** Error running postprocess_batch_list: C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\AI\Forgetest\webui\modules\scripts.py", line 854, in postprocess_batch_list
script.postprocess_batch_list(p, pp, *script_args, **kwargs)
File "C:\AI\Forgetest\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\AI\Forgetest\webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 463, in postprocess_batch_list
self.process_unit_after_every_sampling(p, unit, self.current_params[i], pp, *args, **kwargs)
KeyError: 0

---
Total progress: 100%|████████████████████████████████████████████████████████████████| 150/150 [00:43<00:00, 3.43it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████| 150/150 [00:43<00:00, 1.94it/s]
```

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with extensions-builtin/sd_forge_controlnet/scripts/controlnet.py, especially the reported lines 333, 450, 457, and 463, and reproduce the issue using the fresh-install steps with ControlNet depth. Trace the AssertionError and subsequent KeyError messages; done means ControlNet guidance works without those errors during generation.

Written by the indexing model from the issue text.

Assessment

Tech stack: python
Domain: machine-learning
Issue type: Bug
Difficulty: 4/5
Estimated time: 3-5 days
Activity status: Stale
Clarity: Needs clarification
Newbie friendliness: 25/100

[Bug]: Controlnet intergrated broken after update

Description

Contributor guide

Research direction

Assessment

Get new issues in your inbox