Inconsistency of CLIP-L handling breaks things
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 155
Description
### Expected Behavior
Comfy should adjust according to the actual capabilities of the CLIP model. Currently, it does not, and then breaks in some (but not all) areas due to a hardcode somewhere.
### Actual Behavior
some parts of core comfyUI work fine with LongCLIP-L and others do not
### Steps to Reproduce
Load SD1.5 base model in "load checkpoint'.
Load LongCLIP-L ( https://huggingface.co/zer0int/LongCLIP-GmP-ViT-L-14/resolve/main/Long-ViT-L-14-BEST-GmP-smooth-ft.safetensors?download=true ) seperately via "Load CLIP"
set up standard rendering pipeline. Works fine.
Is clearly rendering with the NEW clip, not the model's builtin one
(I did side-by-side pipelines from the same model checkpopint to compare and prove this)
Set up a "Save Checkpoint" node from the above two nodes. Completes without any visible errors in the GUI.
Now, take the resultant saved checkpoint, point the "load checkpoint" node at it, and try to use.
Blows up because of expected size mismatch in token dimensions.
So ideally, either the loader needs to not have "77 tokens" hardcoded in it... or, if it is reading it from model defs, then the "save checkpoint" node needs to write the actual token count into the checkpoint spec info, instead of the hardcoded 77
### Debug Logs
```powershell
15:12:04.864 [Debug] [ComfyUI-0/STDERR] !!! Exception during processing !!! Error(s) in loading state_dict for SD1ClipModel:
15:12:04.866 [Debug] [ComfyUI-0/STDERR] size mismatch for clip_l.transformer.text_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([248, 768]) from checkpoint, the shape in current model is torch.Size([77, 768]).
15:12:04.869 [Warning] [ComfyUI-0/STDERR] Traceback (most recent call last):
15:12:04.870 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 327, in execute
15:12:04.871 [Warning] [ComfyUI-0/STDERR] output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
15:12:04.872 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.873 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 202, in get_output_data
15:12:04.874 [Warning] [ComfyUI-0/STDERR] return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
15:12:04.876 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.877 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 174, in _map_node_over_list
15:12:04.878 [Warning] [ComfyUI-0/STDERR] process_inputs(input_dict, i)
15:12:04.879 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 163, in process_inputs
15:12:04.881 [Warning] [ComfyUI-0/STDERR] results.append(getattr(obj, func)(**inputs))
15:12:04.883 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.884 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\nodes.py", line 570, in load_checkpoint
15:12:04.885 [Warning] [ComfyUI-0/STDERR] out = comfy.sd.load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, embedding_directory=folder_paths.get_folder_paths("embeddings"))
15:12:04.888 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.890 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\comfy\sd.py", line 905, in load_checkpoint_guess_config
15:12:04.892 [Warning] [ComfyUI-0/STDERR] out = load_state_dict_guess_config(sd, output_vae, output_clip, output_clipvision, embedding_directory, output_model, model_options, te_model_options=te_model_options, metadata=metadata)
15:12:04.893 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.894 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\comfy\sd.py", line 965, in load_state_dict_guess_config
15:12:04.896 [Warning] [ComfyUI-0/STDERR] m, u = clip.load_sd(clip_sd, full_model=True)
15:12:04.901 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.903 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\ComfyUI\comfy\sd.py", line 235, in load_sd
15:12:04.904 [Warning] [ComfyUI-0/STDERR] return self.cond_stage_model.load_state_dict(sd, strict=False)
15:12:04.907 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:12:04.909 [Warning] [ComfyUI-0/STDERR] File "S:\StableSwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 2152, in load_state_dict
15:12:04.911 [Warning] [ComfyUI-0/STDERR] raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
15:12:04.912 [Warning] [ComfyUI-0/STDERR] RuntimeError: Error(s) in loading state_dict for SD1ClipModel:
15:12:04.914 [Warning] [ComfyUI-0/STDERR] size mismatch for clip_l.transformer.text_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([248, 768]) from checkpoint, the shape in current model is torch.Size([77, 768]).
```
### Other
If you dont have time to debug and test a solution, I would be happy to prototype, test, and submit a PR, if you could just point me in the approximate direction of the right code module for this?
Contributor guide
Assessment
This issue has not been assessed yet.