Comfy-Org / Comfy-Org/ComfyUI

Inconsistency of CLIP-L handling breaks things

Open
#7,538 1 comment 0 reactions 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 6h
Merged PRs (30d)
155

Description

### Expected Behavior

Comfy should adjust according to the actual capabilities of the CLIP model. Currently, it does not, and then breaks in some (but not all) areas due to a hardcode somewhere.

### Actual Behavior

some parts of core comfyUI work fine with LongCLIP-L and others do not

### Steps to Reproduce

Load SD1.5 base model in "load checkpoint'.
Load LongCLIP-L ( https://huggingface.co/zer0int/LongCLIP-GmP-ViT-L-14/resolve/main/Long-ViT-L-14-BEST-GmP-smooth-ft.safetensors?download=true ) seperately via "Load CLIP"

set up standard rendering pipeline. Works fine.
Is clearly rendering with the NEW clip, not the model's builtin one
(I did side-by-side pipelines from the same model checkpopint to compare and prove this)

Set up a "Save Checkpoint" node from the above two nodes. Completes without any visible errors in the GUI.

Now, take the resultant saved checkpoint, point the "load checkpoint" node at it, and try to use.

Blows up because of expected size mismatch in token dimensions.

So ideally, either the loader needs to not have "77 tokens" hardcoded in it... or, if it is reading it from model defs, then the "save checkpoint" node needs to write the actual token count into the checkpoint spec info, instead of the hardcoded 77

### Debug Logs

```powershell
15:12:04.864 [Debug] [ComfyUI-0/STDERR] !!! Exception during processing !!! Error(s) in loading state_dict for SD1ClipModel:
15:12:04.866 [Debug] [ComfyUI-0/STDERR] size mismatch for clip_l.transformer.text_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([248, 768]) from checkpoint, the shape in current model is torch.Size([77, 768]).
15:12:04.869 [Warning] [ComfyUI-0/STDERR] Traceback (most recent call last):
15:12:04.870 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 327, in execute
15:12:04.871 [Warning] [ComfyUI-0/STDERR] output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
15:12:04.872 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.873 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 202, in get_output_data
15:12:04.874 [Warning] [ComfyUI-0/STDERR] return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
15:12:04.876 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.877 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 174, in _map_node_over_list
15:12:04.878 [Warning] [ComfyUI-0/STDERR] process_inputs(input_dict, i)
15:12:04.879 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 163, in process_inputs
15:12:04.881 [Warning] [ComfyUI-0/STDERR] results.append(getattr(obj, func)(**inputs))
15:12:04.883 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.884 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\nodes.py", line 570, in load_checkpoint
15:12:04.885 [Warning] [ComfyUI-0/STDERR] out = comfy.sd.load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, embedding_directory=folder_paths.get_folder_paths("embeddings"))
15:12:04.888 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.890 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\comfy\sd.py", line 905, in load_checkpoint_guess_config
15:12:04.892 [Warning] [ComfyUI-0/STDERR] out = load_state_dict_guess_config(sd, output_vae, output_clip, output_clipvision, embedding_directory, output_model, model_options, te_model_options=te_model_options, metadata=metadata)
15:12:04.893 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.894 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\comfy\sd.py", line 965, in load_state_dict_guess_config
15:12:04.896 [Warning] [ComfyUI-0/STDERR] m, u = clip.load_sd(clip_sd, full_model=True)
15:12:04.901 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.903 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\comfy\sd.py", line 235, in load_sd
15:12:04.904 [Warning] [ComfyUI-0/STDERR] return self.cond_stage_model.load_state_dict(sd, strict=False)
15:12:04.907 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.909 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 2152, in load_state_dict
15:12:04.911 [Warning] [ComfyUI-0/STDERR] raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
15:12:04.912 [Warning] [ComfyUI-0/STDERR] RuntimeError: Error(s) in loading state_dict for SD1ClipModel:
15:12:04.914 [Warning] [ComfyUI-0/STDERR] size mismatch for clip_l.transformer.text_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([248, 768]) from checkpoint, the shape in current model is torch.Size([77, 768]).
```

### Other

If you dont have time to debug and test a solution, I would be happy to prototype, test, and submit a PR, if you could just point me in the approximate direction of the right code module for this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.