Comfy-Org / Comfy-Org/ComfyUI

CheckpointSave fails with FakeDevice when Dynamic VRAM is enabled

Open
#14,075 6 comments 3 reactions 1 assignee Claimed by @rattus128 View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

Issue description:
Attempting to save a merged checkpoint (UNet + merged CLIP + VAE) with the CheckpointSave node causes a TypeError deep in diffusers_convert.py because tensors are on a FakeDevice (a result of dynamic offloading) and torch.empty() cannot allocate on a FakeDevice.

Expected behavior:
The checkpoint should save successfully, or ComfyUI should automatically move the tensors to the CPU/GPU before running the concatenation in diffusers_convert.cat_tensors.

### Actual Behavior

Actual behavior:

TypeError: empty() received an invalid combination of arguments - got (list, dtype=torch.dtype, device=FakeDevice), but expected one of:
* (tuple of ints size, *, ...)
Full traceback:

File "...\ComfyUI\comfy\diffusers_convert.py", line 125, in cat_tensors
out = torch.empty(shape, device=tensors[0].device, dtype=tensors[0].dtype)
TypeError: empty() received an invalid combination of arguments ...
The error originates in convert_text_enc_state_dict_v20 when trying to rebuild a combined in_proj_weight from sub‑tensors that reside on a FakeDevice.

### Steps to Reproduce

Start ComfyUI with dynamic VRAM enabled.

Load a base SDXL model via CheckpointLoaderSimple.

Load two additional CLIP models via DualCLIPLoader.

Merge the two CLIP outputs using CLIPMergeSimple.

Connect the UNet, merged CLIP, and VAE to a CheckpointSave node.

Queue the prompt – the error occurs immediately during the save process.

### Debug Logs

```powershell
TypeError: empty() received an invalid combination of arguments - got (list, dtype=torch.dtype, device=FakeDevice), but expected one of:
* (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
* (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)

2026-05-22T20:28:22.846670 - DynamicVRAM support detected and enabled
***
2026-05-22T20:28:53.954264 - got prompt
2026-05-22T20:28:54.089091 - model weight dtype torch.float16, manual cast: None
2026-05-22T20:28:54.096609 - model_type EPS
2026-05-22T20:28:54.133652 - Using pytorch attention in VAE
2026-05-22T20:28:54.134655 - Using pytorch attention in VAE
2026-05-22T20:28:54.221479 - VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
2026-05-22T20:28:54.400149 - CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
2026-05-22T20:28:54.559161 - CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
2026-05-22T20:28:56.032101 - Requested to load SDXLClipModel
2026-05-22T20:28:56.084299 - Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 714 patches attached.
2026-05-22T20:28:56.097337 - Requested to load SDXL
2026-05-22T20:28:56.193015 - Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 714 patches attached.
2026-05-22T20:28:56.273492 - Model SDXL prepared for dynamic VRAM loading. 4897MB Staged. 0 patches attached.
2026-05-22T20:28:56.305699 - !!! Exception during processing !!! empty() received an invalid combination of arguments - got (list, dtype=torch.dtype, device=FakeDevice), but expected one of:
* (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
* (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
***
2026-05-22T20:28:56.313931 - Traceback (most recent call last):
File "...\execution.py", line 535, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(...)
File "...\execution.py", line 335, in get_output_data
return_values = await _async_map_node_over_list(...)
File "...\execution.py", line 309, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "...\execution.py", line 297, in process_inputs
result = f(**inputs)
File "...\nodes_model_merging.py", line 248, in save
save_checkpoint(model, clip=clip, vae=vae, ...)
File "...\nodes_model_merging.py", line 227, in save_checkpoint
comfy.sd.save_checkpoint(output_checkpoint, model, clip, vae, ...)
File "...\sd.py", line 1965, in save_checkpoint
sd = model.state_dict_for_saving(clip_sd, vae_sd, clip_vision_sd)
File "...\model_patcher.py", line 1535, in state_dict_for_saving
return self.model.state_dict_for_saving(...)
File "...\model_base.py", line 361, in state_dict_for_saving
extra_sds.append(self.model_config.process_clip_state_dict_for_saving(clip_state_dict))
File "...\supported_models.py", line 246, in process_clip_state_dict_for_saving
state_dict_g = diffusers_convert.convert_text_enc_state_dict_v20(state_dict, "clip_g")
File "...\diffusers_convert.py", line 177, in convert_text_enc_state_dict_v20
new_state_dict[relabelled_key + ".in_proj_weight"] = cat_tensors(tensors)
File "...\diffusers_convert.py", line 125, in cat_tensors
out = torch.empty(shape, device=tensors[0].device, dtype=tensors[0].dtype)
TypeError: empty() received an invalid combination of arguments - got (list, dtype=torch.dtype, device=FakeDevice), but expected one of:
* (tuple of ints size, *, ...)
* (tuple of ints size, *, ...)
2026-05-22T20:28:56.315926 - Prompt executed in 2.36 seconds
```

### Other

Environment:

ComfyUI version: 0.22.0 (standalone Electron build)

OS: Windows 10/11 (win32)

Python: 3.12.9

PyTorch: 2.8.0+cu128

GPU: NVIDIA GeForce RTX 4070 (12GB VRAM)

Launch arguments (relevant):

--disable-cuda-malloc

--use-pytorch-cross-attention

No explicit VRAM mode set, but logs show comfy-aimdo auto-enabled Dynamic VRAM (DynamicVRAM support detected and enabled)

Active custom nodes: None affecting the core error (only standard CLIPMergeSimple, DualCLIPLoader, CheckpointLoaderSimple, CheckpointSave).

Root cause analysis:

The dynamic VRAM manager (comfy-aimdo) offloads model weights and replaces device objects with FakeDevice to save GPU memory.

When CheckpointSave triggers state_dict_for_saving, the internal conversion functions (process_clip_state_dict_for_saving → convert_text_enc_state_dict_v20) attempt to create new tensors directly on the same device as the source tensors (tensors[0].device), which is FakeDevice.

torch.empty() does not support FakeDevice as a valid device, causing the TypeError.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.