Comfy doesn't seem to offload vram correctly on amd cards

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

Comfy should load the clips into vram and after they are done offload them onto system ram so that the model can fully fit on the vram.

### Actual Behavior

When running flux kontext in Q8 on a 24GB rx7900xtx on windows, comfy loads the clips (t5 Q8 / clip-L) into vram but does not offload the into system ram when the model is loaded, therefore the model dips into shared memory which slows down the generation significantly.

When running the clips fully on the cpu the model fits as normal onto the 24 gb and the generation is as fast as expected.

As for my understanding (correct me if I'm wrong) running the clips on cpu or gpu should not make any difference to vram usage if they are correctly offloaded from vram into ram if they are done. This leads me to believe that they don't get correctly offloaded.

With clips on gpu the sampling takes 104s/it (due to shared memory usage)
With clips on cpu the sampling takes take 7s/it

This same issue already happened with wsl but now its different because its running native so instead of partially loading it goes into shared vram.

https://github.com/comfyanonymous/ComfyUI/issues/7693 (sorry for the duplicate issue now but I wasn't fast enough to appeal the auto closure)

### Steps to Reproduce

The default flux kontext workflow with the Q8 version of flux kontext.

### Debug Logs

```powershell
got prompt
Using split attention in VAE
Using split attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.float32
gguf qtypes: F32 (466), Q8_0 (304), BF16 (10)
model weight dtype torch.bfloat16, manual cast: None
model_type FLUX
Requested to load AutoencodingEngine
loaded completely 16742.625 319.7467155456543 True
gguf qtypes: Q8_0 (169), F32 (50)
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
clip missing: ['text_projection.weight']
Requested to load FluxClipModel_
loaded completely 22293.17744140625 5313.69970703125 True
Requested to load Flux
loaded completely 15945.5750078125 12251.357666015625 True
12%|██████████▌ | 1/8 [01:42<11:58, 102.66s/it]
```

### Other

_No response_