Comfy-Org / Comfy-Org/ComfyUI

Comfy Flux Model Support More CustomNodes abilities Such as IPAdapter

Open
#5,678 0 comments 0 reactions 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Feature Idea

Hi ComfyUI team,
We are expecting that the ComfyUI built-in [Flux](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/ldm/flux/model.py#L36) model would be able to support more plug-in abilities such as IP-Adapter.

More concretely, we are expecting that ComfyUI built-in Flux model would have same abilities (implementation) as another ComfyUI built-in [SDXL](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/model_base.py#L365) Model. The SDXL submodel [UNetModel](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/ldm/modules/diffusionmodules/openaimodel.py#L366) and its submodules now having a forward method accepting an extra parameter called "transformer_options", this extra parameter allows UNetModel and its submodules' accepting IPAdapter weight and more relevant info so that UNet can support IPAdapter computation.
image

image

In Conclusion, we would expecting ComfyUI built-in Flux model's forward method accepting extra parameters to support more plug-in abilities such as IP-Adapter.

Thanks and look forward to your reply.

BR,
ComfyUI user

----------------------
Chinese Version:
你好 ComfyUI 团队,

我们希望ComfyUI内置的 [Flux](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/ldm/flux/model.py#L36) 模型可以支持诸如IPAdapter在内的更多插件功能。

具体来说,我们希望达到的效果,和ComfyUI内置的另一个模型 [SDXL](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/model_base.py#L365) 相同即可。SDXL内部使用的 [UNetModel](https://github.com/comfyanonymous/ComfyUI/blob/898615122f224d24cc4c53a30e67c8b0e70579da/comfy/ldm/modules/diffusionmodules/openaimodel.py#L366) 及其相关的子Module的forward方法会接收额外的参数来支持更多插件能力。在目前的Comfy版本中,UNetModel及子Module的forward接收了额外的 transformer_options 参数,因此IPAdapter的相关信息可以通过该参数传入forward完成计算。
image

image

希望Comfy 内置 Flux 模型也可以像 SDXL 一样,forward 支持传入额外参数以支持诸如IPAdapter的需求。

期待回复。

感谢,
ComfyUI用户

### Existing Solutions

Similar Solution to support IPAdapter:
https://github.com/cubiq/ComfyUI_IPAdapter_plus

However the above implementation rely on an important feature which has already merged into comfyui official repo: The Base Model (SDXL in this example) has an extended forward method accept extra "transformer_options" parameter so we can pass IPAdapter related layers+weight+more info into forward to perform IPAdapter computation.

However as currently comfyui built-in Flux model's forward does not allow extra parameters so we are not able to follow above implementation to support IPAdapter for flux model.

### Other

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.