GGUF LTX2.3 - Dynamic VRAM unloads model and reads from SSD again

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

CLIP/VAE: Gemma 3 12B FP8
Model: LTX 2.3 GGUF Q4KM
Workflow native
RTX 4070, 32GB RAM, Linux

ComfyUI commit #: 77e2ed5e01bcb5eb82e05760f4091d67f7d85a71

On second and third run, I still see the CLIP and LTX model being read from the SSD through dstat. It seems they're not being loaded to RAM? My RAM never exceeds 60% in all runs.

```

154M 4096B
124M 308k
178M 24k
95M 0
99M 0
175M 0
153M 4096B
141M 44k
151M 0
143M 4096B
204M 0
218M 10M
192M 4096B
197M 4096B
200M 60k
191M 0
182M 0
181M 0
200M 4096B
224M 0
194M 0
191M 0
179M 0
188M 0
225M 20k
200M 0
204M 0
196M 0
237M 0
205M 0
195M 20k
150M 0
208M 0
230M 0
211M 0
196M 200k
-dsk/total-
read writ
192M 20k
237M 24k
209M 84k
242M 0
194M 0
207M 8192B
198M 16k
204M 320k
230M 384k
213M 512k
216M 140k
218M 15M
120M 152M
124M 105M
101M 174M
130M 96M
96M 187M
143M 95M
105M 112M
256k 6176k
0 4833k
1408k 5128k
2056k 4800k
1280k 5180k
1920k 4392k
```

### Actual Behavior

.

### Steps to Reproduce

.

### Debug Logs

```powershell
.
```

### Other

_No response_