Comfy-Org / Comfy-Org/ComfyUI

safetensors "file not fully covered": checkpoints with trailing bytes load only when DynamicVRAM is active, and load_torch_file's friendly errors are dead code on safetensors 0.8

Open
#15,602 0 comments 0 reactions 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 6h
Merged PRs (30d)
155

Description

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

1. A safetensors file whose header fully describes its tensor data should load the same way on every install. Whether it loads should not depend on DynamicVRAM, which has nothing to do with parsing.
2. When a safetensors file genuinely cannot be loaded, load_torch_file should produce the explanatory message it was written to produce.

### Actual Behavior

1. Strictness depends on DynamicVRAM.

comfy/utils.py:126-141 chooses between two parsers based on comfy.memory_management.aimdo_enabled:

if comfy.memory_management.aimdo_enabled:
sd, metadata = load_safetensors(ckpt) # ComfyUI's own parser
else:
with safetensors.safe_open(ckpt, framework="pt", device=device.type) as f:

ComfyUI's parser reads the header and walks data_offsets, so bytes after the last tensor are irrelevant to it. The Rust parser rejects the file. So a checkpoint with trailing bytes loads on a default NVIDIA install and fails on the same machine started with --highvram.

2. The friendly error messages never fire on safetensors 0.8.

comfy/utils.py:145-148 matches on "HeaderTooLarge" and "MetadataIncompleteBuffer". safetensors 0.8.0 emits lowercase prose instead, so neither branch is reachable and the raw Rust error propagates via raise e at line 149. Measured on 0.8.0:


condition | message
-- | --
bytes after the last tensor | incomplete metadata, file not fully covered
genuinely truncated file | incomplete metadata, file not fully covered
header length exceeds file | invalid header length
absurd header length | header too large
file shorter than 8 bytes | header too small

Note rows one and two: a file with extra bytes and a file missing bytes produce the identical message. So the intended "check the file size and make sure you have copied/downloaded it correctly" advice would be wrong half the time even if it did fire — and a user whose file has trailing bytes will re-download 12 GB and fail identically. Telling the two apart needs the header's declared end compared against the file size, which string matching cannot do.

### Steps to Reproduce

1. Download MiniMax_H3_Ref2VA_pruned_nvfp4.safetensors from https://huggingface.co/Abiray/Minimax-H3-nvfp4-INT4-INT8-Convrot into models/diffusion_models/. It carries 66 bytes after its tensor data exactly as HuggingFace serves it — no truncation, no bad download:

tensors: 1132
data_start: 116,008
declared_end: 12,528,636,800
file_size: 12,528,636,866
trailing_bytes: 66
The trailing bytes are a marker left by the conversion tool: b'\nL2P_bypass_MiniMax_H3_Ref2VA_pruned_nvfp4.safetensors_1785751127\n'

2. Start ComfyUI normally (NVIDIA, torch >= 2.8) and load it with UNETLoader — it loads.

3. Start with --highvram (or --disable-dynamic-vram, --gpu-only, --novram, --cpu) and load it again — it fails.

enables_dynamic_vram() (comfy/cli_args.py:312) returns False for all five of those flags, and main.py:251 additionally requires NVIDIA and torch >= 2.8. So this also affects every AMD, Intel and Mac user, and anyone on torch < 2.8, on any checkpoint with trailing bytes.

The dependency can be shown without a GPU or a server:

```
import sys; sys.path.insert(0, "/path/to/ComfyUI"); sys.argv = sys.argv[:1]
import comfy.memory_management, comfy.utils
print(comfy.memory_management.aimdo_enabled) # False — the strict path
comfy.utils.load_torch_file("MiniMax_H3_Ref2VA_pruned_nvfp4.safetensors")
```

### Debug Logs

```powershell
aimdo_enabled = False
Traceback (most recent call last):
File "strictcheck.py", line 11, in
sd = comfy.utils.load_torch_file(CKPT)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "ComfyUI\comfy\utils.py", line 149, in load_torch_file
raise e
File "ComfyUI\comfy\utils.py", line 133, in load_torch_file
with safetensors.safe_open(ckpt, framework="pt", device=device.type) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
safetensors._safetensors_rust.SafetensorError: Error while deserializing header:
incomplete metadata, file not fully covered

Note the frame at utils.py:149 — raise e, i.e. neither friendly branch matched.

Environment: ComfyUI master @ a779de4d, safetensors 0.8.0, torch 2.12 cu130, RTX 5070 Ti 16 GB, Windows 11.
```

### Other

Suggested fixes, in the order they look safe:

1. Make the message reachable and accurate. Match on the current strings, and use the header rather than the exception text to decide what to say. load_safetensors already parses the header at comfy/utils.py:94-95, so the test is cheap: if the declared end of tensor data is <= the file size the file is not truncated, and the message can say what is actually true — there are N bytes after the tensor data and this parser is strict about them. This is worth doing regardless of what happens to the parser split, and it is the part that costs users the most time.

2. Fall back on that same condition. When the strict parser fails but the header is readable and the declared range fits inside the file, retry leniently.

3. Use one parser everywhere. Cleanest in principle but I don't think it is free, and it may be why the split exists: load_safetensors imports comfy_aimdo.model_mmap and returns tensors built with torch.frombuffer over a read-only mapping, carrying _comfy_tensor_file_slice and _comfy_tensor_mmap_refs on the storage. That is what the dynamic path wants and probably not what the legacy path wants — read-only storage in particular seems likely to matter for in-place weight patching. I may be missing the reason entirely.

If you would rather not couple the strict path to aimdo at all, a lenient reader is about fifty lines of stdlib and torch — header, data_offsets, frombuffer, reshape. I wrote one and checked it tensor-for-tensor against safetensors.load_file on a checkpoint the strict reader does accept; all 917 tensors matched, dtypes and shapes included.

Related: [#3225](https://github.com/comfyanonymous/ComfyUI/issues/3225) is the same symptom reported in 2024 without a diagnosis.

Contributor guide

Open the contributing guide

Research direction

Start in comfy/utils.py:126-141 and compare the DynamicVRAM parser path with the safetensors.safe_open path used by load_torch_file. Reproduce loading with a safetensors checkpoint containing trailing bytes, then verify that valid files load consistently and genuine failures reach the intended friendly error message.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.