Clip loader not producing expected results. Potentially enforcing CPU usage?

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

I expected to use Clip l and Clip g to produce SDXL generations similar to regular images when using built in clip

### Actual Behavior

however, with dual clip loader and clip g and L. It does not and visual gibberish is all that's produced.

Logs indicated "clip missing: ['text_projection.weight']". I swapped to "Long-ViT-L-14-KO-LITE-HuggingFace-TE-only.safetensors" which squashed that bug but still did not produce a viable image.

### Steps to Reproduce

Very basic test setup using only basic nodes and the clip l and g using a variety of SDXL checkpoints.

### Debug Logs

```powershell
First is the normal built in clip it seems to be going through the GPU:
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cuda:0, dtype: torch.float16
Requested to load SDXL
loaded completely 20115.997177886962 4897.0483474731445 True
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:07<00:00, 2.53it/s]
Requested to load AutoencoderKL
loaded completely 11682.100997924805 159.55708122253418 True
Prompt executed in 21.03 seconds

Next is the Dual Clip loader (It seems to forcing the clip through CPU?, even though its on default):
got prompt
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
clip missing: ['text_projection.weight']
Requested to load SDXLClipModel
loaded completely 15059.34391670227 1560.802734375 True
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:06<00:00, 3.27it/s]
Prompt executed in 9.88 seconds

After Clip "Long-ViT-L-14-KO-LITE-HuggingFace-TE-only.safetensors" swap (still on Cpu):
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load SDXLClipModel
loaded completely 15059.34391670227 1561.05322265625 True
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:06<00:00, 3.11it/s]
Prompt executed in 7.08 seconds
```

### Other

_No response_