lllyasviel / lllyasviel/ControlNet

Error using ControlNet Reference w/ Latent Couple

Open
#429 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
34.1k
Forks
3k
PR merge metrics
No merged PRs in 30d

Description

Bit of an edge case, I'm sure — and likely more so an issue with Latent Couple (I'll report there too), but thought I'd mention it...
Trying to create a 910x512 image using:
Clip Skip: 2
Lora: 1
Steps: 15
CFG Scale: 8
ControlNet - Reference - reference_only - My prompt is more important - Resize and Fill - reference image is 910x512
ControlNet - Canny - canny - Pixel Perfect - ControlNet is more important - Resize and Fill - reference image is 910x512
ControlNet - Depth - depth_midas - Pixel Perfect - CN is more important - Resize and Fill - reference image is 910x512
Latent Couple - 4 sections - reference image is 910x512

I'm able to get this setup to run perfectly, only if:
1) I remove the Latent Couple "AND" parts from the prompt
OR
2) Turn OFF ControNet Reference

Here is the error:

```
Loading preprocessor: reference_only
preprocessor resolution = 512
locon load lora method
0%| | 0/15 [00:00\nAND (giant old craggy stone with waterfalls pouring down it:0.9) (with overgrown moss hanging-vines wildflowers growing on it:1.3)\nAND (giant old tree stumps stone:0.9) (with overgrown clumps of soft green moss and creeping-vines and wildflowers growing on it:1.3) \nAND (giant cluster of vines and flowers:0.9) (with butterflies and humming birds fluttering around it it:1.3)', '(bad-artist:0.25) (EasyNegative:1) (low quality, worst quality:1.3) (text, signature, watermark:1.2) (people, person, structure, building, window, house:1.5), fantasy (fire:1.3)', [], 15, 0, False, False, 1, 1, 8, -1.0, -1.0, 0, 0, 0, False, 512, 910, False, 0.7, 2, 'R-ESRGAN 4x+ Anime6B', 0, 0, 0, 0, '', '', [], 0, '\n

\n
Estimated VRAM usage: 7891.27 MB / 10240 MB (77.06%)
\n
(5679 MB system + 2011.16 MB used)
\n
\n ', False, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1}, {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1}, False, 'MultiDiffusion', False, True, 1024, 1024, 128, 128, 84, 1, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 1536, 128, True, True, True, False, False, '', 0, , , , False, False, 'Matrix', 'Horizontal', 'Mask', 'Prompt', '1,1', '0.2', False, False, False, 'Attention', False, '0', '0', '0.4', None, False, '1:1,1:2,1:2', '0:0,0:0,0:1', '0.2,0.8,0.8', 150, 0.2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, '', 5, 24, 12.5, 1000, '', 'DDIM', 0, 64, 64, '', 64, 7.5, 0.42, 'DDIM', 64, 64, 1, 0, 92, True, True, True, False, False, False, 'midas_v21_small', None, None, False, None, None, False, None, None, False, 50) {}
Traceback (most recent call last):
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\call_queue.py", line 57, in f
res = list(func(*args, **kwargs))
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\txt2img.py", line 57, in txt2img
processed = processing.process_images(p)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\processing.py", line 611, in process_images
res = process_images_inner(p)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\processing.py", line 729, in process_images_inner
samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 293, in process_sample
return process.sample_before_CN_hack(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\processing.py", line 977, in sample
samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 383, in sample
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 257, in launch_sampling
return func()
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 383, in
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 159, in forward
x_out[-uncond.shape[0]:] = self.inner_model(x_in[-uncond.shape[0]:], sigma_in[-uncond.shape[0]:], cond=make_condition_dict([uncond], image_cond_in[-uncond.shape[0]:]))
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in
setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
return self.__orig_func(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
out = self.diffusion_model(x, t, context=cc)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 628, in forward_webui
return forward(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 531, in forward
outer.original_forward(
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 797, in forward
h = module(h, emb, context)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
x = layer(x, context)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 334, in forward
x = block(x, context=context[i])
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 269, in forward
return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 121, in checkpoint
return CheckpointFunction.apply(func, len(inputs), *args)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 136, in forward
output_tensors = ctx.run_function(*ctx.input_tensors)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 664, in hacked_basic_transformer_inner_forward
x = self.attn2(self.norm2(x), context=context) + x
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_hijack_optimizations.py", line 515, in scaled_dot_product_no_mem_attention_forward
return scaled_dot_product_attention_forward(self, x, context, mask)
File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_hijack_optimizations.py", line 490, in scaled_dot_product_attention_forward
k = k_in.view(batch_size, -1, h, head_dim).transpose(1, 2)
RuntimeError: shape '[2, -1, 8, 40]' is invalid for input of size 24640
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the 910x512 generation with ControlNet Reference, Canny, Depth, and Latent Couple using the prompt and settings in the report. Start at modules/processing.py, extensions/sd-webui-controlnet/scripts/batch_hijack.py, and extensions/sd-webui-controlnet/scripts/hook.py, following the traceback into the sampling path. Done means the combined configuration completes without the reported error while the two working alternatives remain functional.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.