Last update broke stuffs and keep getting OOM

### Expected Behavior

Finish generation

### Actual Behavior

Lora and Controlnet would stop mid generation, with an error message or not, as if it is just hanging.
The workflow hasn't changed and it wasn't happening before the update.

### Steps to Reproduce

Update confyui and use a lora or controlnet.
Using following args: --lowvram --preview-method auto --use-split-cross-attention

### Debug Logs

```powershell
model weight dtype torch.bfloat16, manual cast: None
model_type FLUX
ComfyUI\.venv\lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
warnings.warn(
Requested to load FluxClipModel_
Loading 1 new model
loaded completely 0.0 9319.23095703125 True
Requested to load AutoencodingEngine
Loading 1 new model
loaded completely 0.0 159.87335777282715 True
Requested to load InstantXControlNetFluxFormat2
Requested to load Flux
Loading 2 new models
loaded completely 0.0 6298.041015625 True
loaded completely 0.0 12119.472778320312 True

0% 0/30 [00:00