Slowness with lora and control net (any control net model) for Flux

### Expected Behavior

No high VRAM usage, and no extreme slowness with controlnet

### Actual Behavior

Technical details : latest version of comfy ui, 3090, pytorch 2.1 cuda 12.1, windows.

I currently use Comfy UI in production and this is really blocking because using multiple more than 32 rank lora on top of flux is extremely VRAM hungry, and using any control net with ControlNetApplyAdvanced or even the one for SD3/Hyuandit is extremely slow.

Comfy UI is currently not stable with my current configuration (windows is not a choice).

In case using GGUF doesnt help at all since speed is 1.8 times slower and control net support is not working for all models.

### Steps to Reproduce

Technical details : latest version of comfy ui, 3090, pytorch 2.1 cuda 12.1, windows.

Just use any control net, or high rank lora stacked.

### Debug Logs

```powershell
None
```

### Other

None