Is it possible to train LoRas on models merged in ComfyUI?

I recently merged 2 finetuned models, on both which I am able to train LoRas (with Kohya for example). But it's not possible to train LoRas on merged model. My workflow:


Error I'm getting during LoRa training on merged model:
```
File "/content/kohya-trainer/train_network_xl_wrapper.py", line 10, in
trainer.train(args)
File "/content/kohya-trainer/train_network.py", line 213, in train
model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator)
File "/content/kohya-trainer/sdxl_train_network.py", line 34, in load_target_model
) = sdxl_train_util.load_target_model(args, accelerator, sdxl_model_util.MODEL_VERSION_SDXL_BASE_V0_9, weight_dtype)
File "/content/kohya-trainer/library/sdxl_train_util.py", line 33, in load_target_model
) = _load_target_model(
File "/content/kohya-trainer/library/sdxl_train_util.py", line 70, in _load_target_model
) = sdxl_model_util.load_models_from_sdxl_checkpoint(model_version, name_or_path, device, weight_dtype)
File "/content/kohya-trainer/library/sdxl_model_util.py", line 260, in load_models_from_sdxl_checkpoint
info1 = text_model1.load_state_dict(te1_sd)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 2189, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(

RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
Missing key(s) in state_dict: "text_model.embeddings.position_ids".
```

Is there any solution for this or you just simply can't train LoRas on model merges?

I asked the same question on ComfyUI git page, but they redirected me here
https://github.com/comfyanonymous/ComfyUI/issues/3718

Start with the failing load_state_dict call in library/sdxl_model_util.py and the surrounding model-loading path named in the traceback. Compare the merged model's checkpoint contents with the expected CLIPTextModel state, then check whether the linked ComfyUI issue provides relevant context. Done means establishing whether merged models are supported and documenting or implementing a confirmed resolution.

Written by the indexing model from the issue text.