lllyasviel / lllyasviel/stable-diffusion-webui-forge
An issue when the amount of active tokens is exactly 75 in the positive prompt (negative prompt has less)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
Hi!
I've noticed this bug (in the parser?) for some time already, but I only caught the exact conditions for it to occur just now, with 98e0adcc78c1641cc9352bfdad8843bfb49cfc19 commit:
```
Traceback (most recent call last):
File "C:\ComfyUI\Forge\webui\modules_forge\main_thread.py", line 30, in work
self.result = self.func(*self.args, **self.kwargs)
File "C:\ComfyUI\Forge\webui\modules\txt2img.py", line 124, in txt2img_function
processed = processing.process_images(p)
File "C:\ComfyUI\Forge\webui\modules\processing.py", line 842, in process_images
res = process_images_inner(p)
File "C:\ComfyUI\Forge\webui\modules\processing.py", line 962, in process_images_inner
p.setup_conds()
File "C:\ComfyUI\Forge\webui\modules\processing.py", line 1601, in setup_conds
super().setup_conds()
File "C:\ComfyUI\Forge\webui\modules\processing.py", line 505, in setup_conds
self.c = self.get_conds_with_caching(prompt_parser.get_multicond_learned_conditioning, prompts, total_steps, [self.cached_c], self.extra_network_data)
File "C:\ComfyUI\Forge\webui\modules\processing.py", line 474, in get_conds_with_caching
cache[1] = function(shared.sd_model, required_prompts, steps, hires_steps, shared.opts.use_old_scheduling)
File "C:\ComfyUI\Forge\webui\modules\prompt_parser.py", line 262, in get_multicond_learned_conditioning
learned_conditioning = get_learned_conditioning(model, prompt_flat_list, steps, hires_steps, use_old_scheduling)
File "C:\ComfyUI\Forge\webui\modules\prompt_parser.py", line 189, in get_learned_conditioning
conds = model.get_learned_conditioning(texts)
File "C:\ComfyUI\Forge\system\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\ComfyUI\Forge\webui\backend\diffusion_engine\sdxl.py", line 112, in get_learned_conditioning
crossattn=torch.cat([cond_l, cond_g], dim=2),
RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 77 but got size 154 for tensor number 1 in the list.
```
I have 75/75 in the positive prompt, and 18/75 in the negative prompt - I hope it's going to be reproducible. 🙂
I used `ViT-L-14-BEST-smooth-GmP-ft.safetensors` text encoder or CLIP `ponyclip_clip_l_ft35.safetensors` along with VAE, when it happened, but I don't think it caused that - using the TE doesn't change the generated SD15|SDXL output at all. And yes, I put the Forge UI into the ComfyUI folder (should've) changed it to 'AI' folder name or something, since Comfy has its own inside, too).🙂
My OS is Windows 10 on RTX 3090:
```
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: f2.0.1v1.10.1-previous-616-g98e0adcc
Commit hash: 98e0adcc78c1641cc9352bfdad8843bfb49cfc19
Launching Web UI with arguments: --cuda-malloc
Using cudaMallocAsync backend.
Total VRAM 24575 MB, total RAM 65213 MB
pytorch version: 2.3.1+cu121
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3090 : cudaMallocAsync
VAE dtype preferences: [torch.bfloat16, torch.float32] -> torch.bfloat16
CUDA Using Stream: False
C:\ComfyUI\Forge\system\python\lib\site-packages\transformers\utils\hub.py:128: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
Using pytorch cross attention
Using pytorch attention for VAE
ControlNet preprocessor location: C:\ComfyUI\Forge\webui\models\ControlNetPreprocessor
sd-webui-prompt-all-in-one background API service started successfully.
Loading additional modules ... done.
2024-11-11 22:10:09,326 - ControlNet - INFO - ControlNet UI callback registered.
Model selected: {'checkpoint_info': {'filename': 'C:\\ComfyUI\\Forge\\webui\\models\\Stable-diffusion\\PONY\\cashmoneyAnime_v5.safetensors', 'hash': '18f6903c'}, 'additional_modules': ['C:\\ComfyUI\\Forge\\webui\\models\\VAE\\PONY\\ponyStandardVAE_v1.safetensors', 'C:\\ComfyUI\\Forge\\webui\\models\\VAE\\textenc\\ponyclip_clip_l_ft35.safetensors'], 'unet_storage_dtype': None}
Using online LoRAs in FP16: False
Running on local URL: http://127.0.0.1:7860
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported case with 75/75 positive tokens and 18/75 negative tokens, then read webui/modules/prompt_parser.py and webui/backend/diffusion_engine/sdxl.py around the traceback. Confirm the conditioning tensors no longer have mismatched sequence lengths and that this prompt completes without the reported RuntimeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100