Slow generation times in Flux, using loras ( fixed by using GGUF models or XLabs loras )

### Actual Behavior

Using loras with Flux is very slow.
This is independent from the lora size.

But performance is good, using loras, if one of these conditions is met:

- XLabs loras are used

- GUFF models are used ( even the larger Q8_0 )

- `--reserve-vram 1.2` is used - AND - only a single lora is used

---

More details, in the previous thread : https://github.com/comfyanonymous/ComfyUI/issues/4618
Initially, I thought this issue was related to lora size, but this is not the case. 
These conditions are independent from lora size:
For instance, `--reserve-vram 1.2` will work for one 1.28 GB lora, but not for two 19 MB loras.

### Other

- GTX 1070 ( 8GB )
- 32 GB RAM
- Windows 10
- pytorch version: 2.3.1+cu121