High lora vram usage after update

### Expected Behavior

Lora should load with minimal vram overhead (considering it is a small lora; the 4/4 rank one is 40mb).

### Actual Behavior

Large vram usage increase when loading certain loras, in my case trained with ai-toolkit by Ostris. When using fp8_e4m3fn flux.dev, vram usage is 14.3gb, regardless of lora size (4/4, 16/16, 32/32). However, downloaded loras made with SimpleTuner only go up to 12.2gb usage. This is a problem when loading the fp16 model because it no longer fits on a 24gb vram GPU.

### Steps to Reproduce

Load a lora made with ai-toolkit by Ostris

### Debug Logs

```powershell
Nothing special to report, but here is a log:

P:\text2image\ComfyUI_windows_portable_v008\ComfyUI_windows_portable>.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --front-end-version Comfy-Org/ComfyUI_frontend@latest --lowvram

Prestartup times for custom nodes:
0.0 seconds: P:\text2image\ComfyUI_windows_portable_v008\ComfyUI_windows_portable\ComfyUI\custom_nodes\rgthree-comfy

Total VRAM 24564 MB, total RAM 97415 MB
pytorch version: 2.3.1+cu121
Set vram state to: LOW_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
Using pytorch cross attention
[Prompt Server] web root: P:\text2image\ComfyUI_windows_portable_v008\ComfyUI_windows_portable\ComfyUI\web_custom_versions\Comfy-Org_ComfyUI_frontend\1.2.20
### Loading: ComfyUI-Impact-Pack (V6.2)
### Loading: ComfyUI-Impact-Pack (Subpack: V0.6)
[WARN] ComfyUI-Impact-Pack: `ComfyUI` or `ComfyUI-Manager` is an outdated version.
[Impact Pack] Wildcards loading done.

[rgthree] Loaded 42 magnificent nodes.
[rgthree] Will use rgthree's optimized recursive execution.

Import times for custom nodes:
0.0 seconds: P:\text2image\ComfyUI_windows_portable_v008\ComfyUI_windows_portable\ComfyUI\custom_nodes\websocket_image_save.py
0.0 seconds: P:\text2image\ComfyUI_windows_portable_v008\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-MultiGPU-main
0.0 seconds: P:\text2image\ComfyUI_windows_portable_v008\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_UltimateSDUpscale
0.0 seconds: P:\text2image\ComfyUI_windows_portable_v008\ComfyUI_windows_portable\ComfyUI\custom_nodes\rgthree-comfy
0.5 seconds: P:\text2image\ComfyUI_windows_portable_v008\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack

Starting server

To see the GUI go to: http://127.0.0.1:8188
got prompt
[rgthree] Using rgthree's optimized recursive execution.
[rgthree] First run patching recursive_output_delete_if_changed and recursive_will_execute.
[rgthree] Note: If execution seems broken due to forward ComfyUI changes, you can disable the optimization from rgthree settings in ComfyUI.
model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
model_type FLUX
Requested to load FluxClipModel_
Loading 1 new model
clip missing: ['text_projection.weight']
Requested to load Flux
Loading 1 new model
0%| | 0/50 [00:00