Lora does not work for flux gguf

### Expected Behavior

Lora should work correctly for flux gguf

### Actual Behavior

Lora does not work correctly for flux gguf

### Steps to Reproduce

Using GGUF FLux checkpoint with any flux lora. The lora does not work. It only works if we do not use gguf

### Debug Logs

```powershell
no error log
```

### Other

_No response_