Flux LoRA loading failure

### Describe the bug

When loading in [this model](https://civitai.com/models/631986) you get a `ValueError(f"Incompatible keys detected: \n\n {', '.join(remaining_keys)}")` error

It looks like it's a LoRA type that doesn't fall into any of the classic conversion script paths.

It looks like a Flux diffusers checkpoint merged with an xlabs lora? I can't really tell.

There aren't a lot of scripts out there to convert this format, but it does seem to get processed properly by ComfyUI.

https://huggingface.co/city96/FLUX.1-dev-gguf/discussions/13

If this is a good starting point, the prefix is slightly different too, since there are scripts that look for "model.diffusion_model." but not just raw "diffusion_model.".

```py

def convert_flux_transformer_checkpoint_to_diffusers(checkpoint, **kwargs):
converted_state_dict = {}
keys = list(checkpoint.keys())
for k in keys:
if "model.diffusion_model." in k:
checkpoint[k.replace("model.diffusion_model.", "")] = checkpoint.pop(k)

if "diffusion_model." in k:
checkpoint[k.replace("diffusion_model.", "")] = checkpoint.pop(k)

```

### Reproduction

Download the safetensors and load them in as a LoRA in a flux pipeline

```
downloaded_file = "pytorch.safetensors"
pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev ", torch_dtype=torch.bfloat16)
pipe.load_lora_weights(downloaded_file)
```

### Logs

```shell
ValueError: Incompatible keys detected:

diffusion_model.double_blocks.0.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.0.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.0.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.0.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.0.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.0.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.0.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.0.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.1.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.1.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.1.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.1.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.1.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.1.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.1.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.1.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.10.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.10.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.10.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.10.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.10.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.10.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.10.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.10.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.11.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.11.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.11.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.11.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.11.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.11.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.11.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.11.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.12.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.12.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.12.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.12.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.12.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.12.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.12.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.12.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.13.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.13.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.13.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.13.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.13.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.13.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.13.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.13.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.14.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.14.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.14.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.14.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.14.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.14.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.14.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.14.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.15.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.15.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.15.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.15.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.15.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.15.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.15.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.15.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.16.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.16.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.16.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.16.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.16.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.16.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.16.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.16.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.17.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.17.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.17.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.17.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.17.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.17.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.17.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.17.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.18.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.18.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.18.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.18.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.18.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.18.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.18.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.18.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.2.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.2.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.2.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.2.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.2.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.2.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.2.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.2.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.3.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.3.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.3.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.3.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.3.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.3.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.3.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.3.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.4.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.4.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.4.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.4.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.4.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.4.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.4.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.4.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.5.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.5.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.5.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.5.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.5.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.5.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.5.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.5.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.6.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.6.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.6.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.6.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.6.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.6.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.6.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.6.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.7.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.7.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.7.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.7.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.7.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.7.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.7.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.7.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.8.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.8.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.8.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.8.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.8.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.8.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.8.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.8.txt_attn.qkv.lora_up.weight, diffusion_model.double_blocks.9.img_attn.proj.lora_down.weight, diffusion_model.double_blocks.9.img_attn.proj.lora_up.weight, diffusion_model.double_blocks.9.img_attn.qkv.lora_down.weight, diffusion_model.double_blocks.9.img_attn.qkv.lora_up.weight, diffusion_model.double_blocks.9.txt_attn.proj.lora_down.weight, diffusion_model.double_blocks.9.txt_attn.proj.lora_up.weight, diffusion_model.double_blocks.9.txt_attn.qkv.lora_down.weight, diffusion_model.double_blocks.9.txt_attn.qkv.lora_up.weight
```

### System Info

- 🤗 Diffusers version: 0.31.0
- Platform: macOS-15.0.1-arm64-arm-64bit
- Running on Google Colab?: No
- Python version: 3.11.10
- PyTorch version (GPU?): 2.5.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.26.2
- Transformers version: 4.46.0
- Accelerate version: 0.34.2
- PEFT version: 0.13.2
- Bitsandbytes version: 0.42.0
- Safetensors version: 0.4.5
- xFormers version: 0.0.29+68b7fd1.d20241024
- Accelerator: Apple M2 Max

### Who can help?

@sayakpaul