Add support for GLIGEN-XL (SDXL supported GLIGEN)
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Feature Idea
There is a recent GLIGEN model updated for SDXL 1.0 linked below. I have tried for a few days now to get it working in ComfyUI however I have been unsuccessful. If someone with more experience could implement support for this model or point me in the right direction on getting it working it would be much appreciated.
**IGLIGEN-XL:** [https://github.com/jiuntian/igligen-xl](https://github.com/jiuntian/igligen-xl)
**GLIGEN-XL-512 model:** [https://huggingface.co/jiuntian/gligen-xl-512](https://huggingface.co/jiuntian/gligen-xl-512)
**GLIGEN-XL-1024 model:** [https://huggingface.co/jiuntian/gligen-xl-1024](https://huggingface.co/jiuntian/gligen-xl-1024)
I am using the example GLIGEN workflow in ComfyUI for my tests. I have tried combining the safetensors shards in the unet folder into a single safetensor (referring to the huggingface repo) and loaded it in ComfyUI using the GLIGENLoader. Then loaded the sd_xl_base_1.0 checkpoint. However I get the error when running the workflow:
`Sizes of tensors must match except in dimension 0. Expected size 1280 but got size 768 for tensor number 2 in the list.`
I suspect ComfyUI is coded only to support GLIGEN for SD1.4. I couldn't work out what would need to be changed in the code to support this new version of GLIGEN. I tried changing `key_dim = 768` in gligen.py in `load_gligen() `to `key_dim = 1280 ` but that just produces a new error:
`The size of tensor a (1280) must match the size of tensor b (2048) at non-singleton dimension 2`
Like I said I don't know enough about all this to properly implement it my self. Hopefully someone can take a look.
### Existing Solutions
_No response_
### Other
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.