DynamicVRAM breaks loading models

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

Loading models works correctly and prompts are handled correctly

### Actual Behavior

An error occurs when trying to load the checkpoint and encode the prompt:
```
!!! Exception during processing !!! Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__index_select)
```

### Steps to Reproduce

Use the following `start_comfyui.bat`:
`.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --port 19000 --output-directory E:\AI\stable-diffusion\Outputs`

OS: Windows 11 Pro (25H2, Build: 26200.7840)
CPU: AMD Ryzen 9 9950X3D
GPU: NVIDIA RTX 5090

The below debug log is from my workflow with custom nodes, however removing the custom nodes and using a simple Debug workflow (attached) also produces the same result

[Debug.json](https://github.com/user-attachments/files/25767583/Debug.json)

### Debug Logs

```powershell
got prompt
model weight dtype torch.float16, manual cast: None
model_type V_PREDICTION
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
CLIP: []
Requested to load SDXLClipModel
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
!!! Exception during processing !!! Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__index_select)
Traceback (most recent call last):
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\execution.py", line 524, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\execution.py", line 333, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\execution.py", line 307, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\execution.py", line 295, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\nodes.py", line 80, in encode
return (clip.encode_from_tokens_scheduled(tokens), )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd.py", line 313, in encode_from_tokens_scheduled
pooled_dict = self.encode_from_tokens(tokens, return_pooled=return_pooled, return_dict=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd.py", line 377, in encode_from_tokens
o = self.cond_stage_model.encode_token_weights(tokens)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sdxl_clip.py", line 59, in encode_token_weights
g_out, g_pooled = self.clip_g.encode_token_weights(token_weight_pairs_g)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd1_clip.py", line 45, in encode_token_weights
o = self.encode(to_encode)
^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd1_clip.py", line 306, in encode
return self(tokens)
^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1776, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1787, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd1_clip.py", line 266, in forward
embeds, attention_mask, num_tokens, embeds_info = self.process_tokens(tokens, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd1_clip.py", line 213, in process_tokens
tokens_embed = self.transformer.get_input_embeddings()(tokens_embed, out_dtype=torch.float32)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1776, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1787, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\ops.py", line 552, in forward
return self.forward_comfy_cast_weights(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\ops.py", line 544, in forward_comfy_cast_weights
x = torch.nn.functional.embedding(input, weight, self.padding_idx, self.max_norm, self.norm_type, self.scale_grad_by_freq, self.sparse).to(dtype=output_dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\stable-diffusion\UI\comfy-ui\python_embeded\Lib\site-packages\torch\nn\functional.py", line 2567, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__index_select)

Prompt executed in 3.42 seconds
```

### Other

Full debug logs attached (broken.log). Testing with `--disable-dynamic-vram` does work, log is also attached (working.log)

[working.log](https://github.com/user-attachments/files/25767646/working.log)
[broken.log](https://github.com/user-attachments/files/25767647/broken.log)