KeyError when loading LoRA for Flux model: missing lora_unet_final_layer_adaLN_modulation_1 weights

I'm trying to run Overlay-Kontext-Dev-LoRA locally by loading the LoRA weights using the pipe.load_lora_weights() function. However, I encountered the following error during execution:

> KeyError: 'lora_unet_final_layer_adaLN_modulation_1.lora_down.weight'

```
import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image

Load the pipeline with a specific torch data type for GPU optimization
pipe = DiffusionPipeline.from_pretrained(
"black-forest-labs/FLUX.1-Kontext-dev",
torch_dtype=torch.bfloat16
)

Move the entire pipeline to the GPU
pipe.to("cuda")

Load LoRA weights (this will also be on the GPU)
pipe.load_lora_weights("ilkerzgi/Overlay-Kontext-Dev-LoRA")

prompt = "Place it"
input_image = load_image("img2.png")

The pipeline will now run on the GPU
image = pipe(image=input_image, prompt=prompt).images[0]

image.save("output_image.png")
```

Environment:
diffusers version: 0.35.0.dev0
Python: 3.10
Running locally on a ubuntu environment with RTX 4090

> Additional Note:
> The model file size is also quite large. I may need to quantize it before running it on the 4090 to avoid out-of-memory issues.
>
> Would appreciate any help or suggestions on how to resolve the loading issue. Thank you!

Start by reproducing the provided Python script with diffusers 0.35.0.dev0 and inspect the pipe.load_lora_weights() entry point using the Overlay-Kontext-Dev-LoRA and FLUX.1-Kontext-dev model. Compare the LoRA keys with the pipeline's expected keys, and consider the issue done when the weights load without the missing lora_unet_final_layer_adaLN_modulation_1 error.

Scritto dal modello di indicizzazione a partire dal testo della issue.