Comfy Flux Model Support More CustomNodes abilities Such as IPAdapter
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Feature Idea
Hi ComfyUI team,
We are expecting that the ComfyUI built-in [Flux](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/ldm/flux/model.py#L36) model would be able to support more plug-in abilities such as IP-Adapter.
More concretely, we are expecting that ComfyUI built-in Flux model would have same abilities (implementation) as another ComfyUI built-in [SDXL](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/model_base.py#L365) Model. The SDXL submodel [UNetModel](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/ldm/modules/diffusionmodules/openaimodel.py#L366) and its submodules now having a forward method accepting an extra parameter called "transformer_options", this extra parameter allows UNetModel and its submodules' accepting IPAdapter weight and more relevant info so that UNet can support IPAdapter computation.
In Conclusion, we would expecting ComfyUI built-in Flux model's forward method accepting extra parameters to support more plug-in abilities such as IP-Adapter.
Thanks and look forward to your reply.
BR,
ComfyUI user
----------------------
Chinese Version:
你好 ComfyUI 团队,
我们希望ComfyUI内置的 [Flux](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/ldm/flux/model.py#L36) 模型可以支持诸如IPAdapter在内的更多插件功能。
具体来说,我们希望达到的效果,和ComfyUI内置的另一个模型 [SDXL](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/model_base.py#L365) 相同即可。SDXL内部使用的 [UNetModel](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/ldm/modules/diffusionmodules/openaimodel.py#L366) 及其相关的子Module的forward方法会接收额外的参数来支持更多插件能力。在目前的Comfy版本中,UNetModel及子Module的forward接收了额外的 transformer_options 参数,因此IPAdapter的相关信息可以通过该参数传入forward完成计算。
希望Comfy 内置 Flux 模型也可以像 SDXL 一样,forward 支持传入额外参数以支持诸如IPAdapter的需求。
期待回复。
感谢,
ComfyUI用户
### Existing Solutions
Similar Solution to support IPAdapter:
https://github.com/cubiq/ComfyUI_IPAdapter_plus
However the above implementation rely on an important feature which has already merged into comfyui official repo: The Base Model (SDXL in this example) has an extended forward method accept extra "transformer_options" parameter so we can pass IPAdapter related layers+weight+more info into forward to perform IPAdapter computation.
However as currently comfyui built-in Flux model's forward does not allow extra parameters so we are not able to follow above implementation to support IPAdapter for flux model.
### Other
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.