IP-Adapter cannot work in the forge, it can only produces some abstract color blocks.

Nobody has claimed this yet.

IIB Database file has been successfully backed up to the backup folder.
Startup time: 61.4s (prepare environment: 12.7s, import torch: 20.1s, other imports: 0.9s, load scripts: 9.1s, initialize google blockly: 5.9s, create ui: 7.0s, gradio launch: 5.0s, app_started_callback: 0.5s).
Environment vars changed: {'stream': False, 'inference_memory': 1024.0, 'pin_shared_memory': False}
[GPU Setting] You will use 93.22% GPU memory (14078.00 MB) to load weights, and use 6.78% GPU memory (1024.00 MB) to do matrix computation.
Loading Model: {'checkpoint_info': {'filename': '/content/stable-diffusion-webui-forge/models/Stable-diffusion/noobaiXLNAIXL_epsilonPred11Version.safetensors', 'hash': '1ce6b882'}, 'additional_modules': ['/content/stable-diffusion-webui-forge/models/VAE/sdxl.vae.safetensors'], 'unet_storage_dtype': None}
[Unload] Trying to free all memory for cuda:0 with 0 models keep loaded ... Done.
StateDict Keys: {'unet': 1680, 'vae': 250, 'text_encoder': 196, 'text_encoder_2': 518, 'ignore': 0}
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
IntegratedAutoencoderKL Unexpected: ['model_ema.decay', 'model_ema.num_updates']
K-Model Created: {'storage_dtype': torch.float16, 'computation_dtype': torch.float16}
Model loaded in 38.0s (unload existing model: 0.7s, forge model load: 37.3s).
2024-12-11 11:49:42,099 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
2024-12-11 11:49:42,164 - ControlNet - INFO - Using preprocessor: None
2024-12-11 11:49:42,165 - ControlNet - INFO - preprocessor resolution = 512
*** Error running process: /content/stable-diffusion-webui-forge/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py
Traceback (most recent call last):
File "/content/stable-diffusion-webui-forge/modules/scripts.py", line 844, in process
script.process(p, *script_args)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/content/stable-diffusion-webui-forge/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py", line 554, in process
self.process_unit_after_click_generate(p, unit, params, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/content/stable-diffusion-webui-forge/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py", line 411, in process_unit_after_click_generate
assert unit.model != 'None', 'You have not selected any control model!'
AssertionError: You have not selected any control model!

---
[Unload] Trying to free 3051.58 MB for cuda:0 with 0 models keep loaded ... Done.
[Memory Management] Target: JointTextEncoder, Free GPU: 9979.66 MB, Model Require: 1559.68 MB, Previously Loaded: 0.00 MB, Inference Require: 1024.00 MB, Remaining: 7395.98 MB, All loaded to GPU.
Moving model(s) has taken 0.51 seconds
[Unload] Trying to free 1024.00 MB for cuda:0 with 1 models keep loaded ... Current free memory is 8178.86 MB ... Done.
*** Error running process_before_every_sampling: /content/stable-diffusion-webui-forge/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py
Traceback (most recent call last):
File "/content/stable-diffusion-webui-forge/modules/scripts.py", line 892, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/content/stable-diffusion-webui-forge/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py", line 561, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
KeyError: 0

---
[Unload] Trying to free 2856.18 MB for cuda:0 with 0 models keep loaded ... Current free memory is 8178.01 MB ... Done.
[Memory Management] Target: KModel, Free GPU: 8178.01 MB, Model Require: 0.00 MB, Previously Loaded: 4897.05 MB, Inference Require: 1024.00 MB, Remaining: 7154.01 MB, All loaded to GPU.
Moving model(s) has taken 0.04 seconds
100% 28/28 [00:22<00:00, 1.23it/s]
[Unload] Trying to free 8820.57 MB for cuda:0 with 0 models keep loaded ... Current free memory is 8167.51 MB ... Unload model JointTextEncoder Current free memory is 9931.70 MB ... Done.
[Memory Management] Target: IntegratedAutoencoderKL, Free GPU: 9931.70 MB, Model Require: 319.11 MB, Previously Loaded: 0.00 MB, Inference Require: 1024.00 MB, Remaining: 8588.58 MB, All loaded to GPU.
Moving model(s) has taken 1.33 seconds
*** Error running postprocess_batch_list: /content/stable-diffusion-webui-forge/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py
Traceback (most recent call last):
File "/content/stable-diffusion-webui-forge/modules/scripts.py", line 916, in postprocess_batch_list
script.postprocess_batch_list(p, pp, *script_args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/content/stable-diffusion-webui-forge/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py", line 567, in postprocess_batch_list
self.process_unit_after_every_sampling(p, unit, self.current_params[i], pp, *args, **kwargs)
KeyError: 0

---
2024-12-11 11:50:23,567 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
2024-12-11 11:50:23,598 - ControlNet - INFO - Using preprocessor: InsightFace+CLIP-H (IPAdapter)
2024-12-11 11:50:23,598 - ControlNet - INFO - preprocessor resolution = 0.5
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /content/stable-diffusion-webui-forge/models/insightface/models/buffalo_l/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /content/stable-diffusion-webui-forge/models/insightface/models/buffalo_l/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /content/stable-diffusion-webui-forge/models/insightface/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /content/stable-diffusion-webui-forge/models/insightface/models/buffalo_l/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /content/stable-diffusion-webui-forge/models/insightface/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)
2024-12-11 11:50:39,949 - ControlNet - INFO - Current ControlNet IPAdapterPatcher: /content/stable-diffusion-webui-forge/models/ControlNet/ip-adapter-plus_sdxl_vit-h.safetensors
[Unload] Trying to free 3302.87 MB for cuda:0 with 0 models keep loaded ... Current free memory is 9611.95 MB ... Done.
[Memory Management] Target: JointTextEncoder, Free GPU: 9611.95 MB, Model Require: 1752.98 MB, Previously Loaded: 0.00 MB, Inference Require: 1024.00 MB, Remaining: 6834.97 MB, All loaded to GPU.
Moving model(s) has taken 0.60 seconds
[Unload] Trying to free 1024.00 MB for cuda:0 with 1 models keep loaded ... Current free memory is 7846.25 MB ... Done.
INFO: the IPAdapter reference image is not a square, CLIPImageProcessor will resize and crop it at the center. If the main focus of the picture is not in the middle the result might not be what you are expecting.
[Unload] Trying to free 2591.27 MB for cuda:0 with 0 models keep loaded ... Current free memory is 7845.40 MB ... Done.
[Memory Management] Target: CLIPVisionModelWithProjection, Free GPU: 7845.40 MB, Model Require: 1205.59 MB, Previously Loaded: 0.00 MB, Inference Require: 1024.00 MB, Remaining: 5615.81 MB, All loaded to GPU.
Moving model(s) has taken 0.39 seconds
[Unload] Trying to free 1024.00 MB for cuda:0 with 1 models keep loaded ... Current free memory is 6638.74 MB ... Done.
2024-12-11 11:50:49,038 - ControlNet - INFO - ControlNet Method InsightFace+CLIP-H (IPAdapter) patched.
[Unload] Trying to free 2856.18 MB for cuda:0 with 0 models keep loaded ... Current free memory is 5827.50 MB ... Done.
[Memory Management] Target: KModel, Free GPU: 5827.50 MB, Model Require: 0.00 MB, Previously Loaded: 4897.05 MB, Inference Require: 1024.00 MB, Remaining: 4803.50 MB, All loaded to GPU.
Moving model(s) has taken 0.03 seconds
100% 28/28 [00:21<00:00, 1.27it/s]
[Unload] Trying to free 8405.72 MB for cuda:0 with 1 models keep loaded ... Current free memory is 5827.02 MB ... Unload model JointTextEncoder Current free memory is 7590.83 MB ... Unload model CLIPVisionModelWithProjection Current free memory is 8797.49 MB ... Done.
Memory cleanup has taken 2.66 seconds
2024-12-11 11:54:04,388 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
2024-12-11 11:54:04,424 - ControlNet - INFO - Using preprocessor: InsightFace+CLIP-H (IPAdapter)
2024-12-11 11:54:04,424 - ControlNet - INFO - preprocessor resolution = 0.5
2024-12-11 11:54:04,429 - ControlNet - INFO - Current ControlNet IPAdapterPatcher: /content/stable-diffusion-webui-forge/models/ControlNet/ip-adapter-plus_sdxl_vit-h.safetensors
INFO: the IPAdapter reference image is not a square, CLIPImageProcessor will resize and crop it at the center. If the main focus of the picture is not in the middle the result might not be what you are expecting.
[Unload] Trying to free 2591.27 MB for cuda:0 with 0 models keep loaded ... Current free memory is 8797.49 MB ... Done.
[Memory Management] Target: CLIPVisionModelWithProjection, Free GPU: 8797.49 MB, Model Require: 1205.59 MB, Previously Loaded: 0.00 MB, Inference Require: 1024.00 MB, Remaining: 6567.90 MB, All loaded to GPU.
Moving model(s) has taken 0.49 seconds
[Unload] Trying to free 1024.00 MB for cuda:0 with 1 models keep loaded ... Current free memory is 7591.63 MB ... Done.
2024-12-11 11:54:09,023 - ControlNet - INFO - ControlNet Method InsightFace+CLIP-H (IPAdapter) patched.
[Unload] Trying to free 2856.18 MB for cuda:0 with 0 models keep loaded ... Current free memory is 7591.00 MB ... Done.
[Memory Management] Target: KModel, Free GPU: 7591.00 MB, Model Require: 0.00 MB, Previously Loaded: 4897.05 MB, Inference Require: 1024.00 MB, Remaining: 6567.00 MB, All loaded to GPU.
Moving model(s) has taken 0.06 seconds
100% 28/28 [00:21<00:00, 1.28it/s]
[Unload] Trying to free 8405.72 MB for cuda:0 with 1 models keep loaded ... Current free memory is 7590.52 MB ... Unload model CLIPVisionModelWithProjection Current free memory is 8796.38 MB ... Done.
Memory cleanup has taken 0.77 seconds

Start with extensions-builtin/sd_forge_controlnet/scripts/controlnet.py, especially process_unit_after_click_generate, process_before_every_sampling, and postprocess_batch_list. Reproduce the IP-Adapter flow using the logged SDXL model, ControlNet model, InsightFace+CLIP-H preprocessor, and reference image. Done means the generation completes without the reported AssertionError or KeyError and the output is no longer abstract color blocks.

Written by the indexing model from the issue text.