lllyasviel / lllyasviel/stable-diffusion-webui-forge
Error Using v1.5 Model with ControlNet Tile Controlnet Mode Set and Multidiffusion with Tile Batch Size at the Same Time
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
### Repro steps using integrated ControlNet and Multidiffusion:
1. load a v1.5 model in `sd` mode into img2img
2. enable ControlNet node 0 (tile, `control_v11f1e_sd15_tile [a371b31b]`, `My prompt is more important` or `ControlNet is more important`)
3. enable Multiduffusion (batch size > 1)
### Result
`RuntimeError: The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 0`
```
2024-10-31 14:13:16,316 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
2024-10-31 14:13:16,426 - ControlNet - INFO - Using preprocessor: tile_resample
2024-10-31 14:13:16,426 - ControlNet - INFO - preprocessor resolution = 1536
2024-10-31 14:13:16,447 - ControlNet - INFO - Current ControlNet ControlNetPatcher: /home/rt/ai/models/stable-diffusion/ControlNet/v1.5/control_v11f1e_sd15_tile.pth
[Unload] Trying to free 14163.61 MB for cuda:0 with 1 models keep loaded ... Current free memory is 17865.67 MB ... Done.
2024-10-31 14:13:16,941 - ControlNet - INFO - ControlNet Method tile_resample patched.
[Unload] Trying to free 6009.88 MB for cuda:0 with 0 models keep loaded ... Current free memory is 17960.11 MB ... Done.
[Memory Management] Target: KModel, Free GPU: 17960.11 MB, Model Require: 0.00 MB, Previously Loaded: 1639.41 MB, Inference Require: 1021.00 MB, Remaining: 16939.11 MB, All loaded to GPU.
[Memory Management] Target: ControlNet, Free GPU: 17960.11 MB, Model Require: 0.00 MB, Previously Loaded: 689.09 MB, Inference Require: 1021.00 MB, Remaining: 16939.11 MB, All loaded to GPU.
Moving model(s) has taken 0.01 seconds
0%| | 0/31 [00:00
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "/home/rt/ai/repos/stable-diffusion-webui-forge/venv/lib64/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/k_diffusion/sampling.py", line 595, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/venv/lib64/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/venv/lib64/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/modules/sd_samplers_cfg_denoiser.py", line 199, in forward
denoised, cond_pred, uncond_pred = sampling_function(self, denoiser_params=denoiser_params, cond_scale=cond_scale, cond_composition=cond_composition)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/backend/sampling/sampling_function.py", line 362, in sampling_function
denoised, cond_pred, uncond_pred = sampling_function_inner(model, x, timestep, uncond, cond, cond_scale, model_options, seed, return_full=True)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/backend/sampling/sampling_function.py", line 303, in sampling_function_inner
cond_pred, uncond_pred = calc_cond_uncond_batch(model, cond, uncond_, x, timestep, model_options)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/backend/sampling/sampling_function.py", line 271, in calc_cond_uncond_batch
output = model_options['model_function_wrapper'](model.apply_model, {"input": input_x, "timestep": timestep_, "c": c, "cond_or_uncond": cond_or_uncond}).chunk(batch_chunks)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/venv/lib64/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/extensions-builtin/sd_forge_multidiffusion/lib_multidiffusion/tiled_diffusion.py", line 457, in __call__
c_tile['control'] = c_in['control_model'].get_control(x_tile, ts_tile, c_tile, len(cond_or_uncond))
File "/home/rt/ai/repos/stable-diffusion-webui-forge/backend/patcher/controlnet.py", line 339, in get_control
return self.control_merge(None, control, control_prev, output_dtype)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/backend/patcher/controlnet.py", line 262, in control_merge
out = compute_controlnet_weighting(out, self)
File "/home/rt/ai/repos/stable-diffusion-webui-forge/backend/patcher/controlnet.py", line 148, in compute_controlnet_weighting
control[k][i] = control_signal * final_weight[:, None, None, None]
RuntimeError: The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 0
The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 0
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the v1.5 img2img case with ControlNet tile mode and Multidiffusion batch size greater than 1, then trace the call from extensions-builtin/sd_forge_multidiffusion/lib_multidiffusion/tiled_diffusion.py through backend/patcher/controlnet.py. Inspect compute_controlnet_weighting and get_control to understand the reported batch-size mismatch. Done means the same configuration completes without the RuntimeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100