Comfy-Org / Comfy-Org/ComfyUI

Access violation (0xC0000005) loading NVFP4-quantized text encoder/CLIP files

Open
#15,397 5 comments 1 reaction 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

The text-encoder/CLIP file loads successfully, or — if something about the file is genuinely incompatible — ComfyUI raises a handled Python exception with a clear error message, instead of crashing the entire process with a native access violation.

### Actual Behavior

ComfyUI terminates with a native access violation (Windows exit code `3221225477` / `0xC0000005`) while loading certain quantized CLIP/text-encoder `.safetensors` files that use the new mixed-precision quantization metadata format (log line: `Found quantization metadata version 1`, `Detected mixed precision quantization`). The crash is reproducible at the same point on every run.

This has been observed across **two unrelated model families** (MiniMax H3 and LTX-2), in both cases while loading a text-encoder file that uses the **NVFP4** quantization format — which the startup log explicitly lists under *emulated ops*, as opposed to *native ops* like `int8_tensorwise` / `convrot_w4a4` / `float8_e4m3fn`.

### Steps to Reproduce

## Environment

- **OS:** Windows 11
- **GPU:** AMD Radeon RX 9060 XT, 16 GB VRAM (gfx1200, RDNA4)
- **GPU driver:** 32.0.31035.1003 (24.07.2026)
- **CPU:** AMD Ryzen 7 7700
- **System RAM:** ~31 GB
- **PyTorch backend:** ROCm/HIP
- **ComfyUI install method:** Comfy Desktop (multi-instance manager)
- **Python:** 3.12 (`standalone-env`), torch installed in per-instance `.venv`
- **Launch arguments tested:** `--disable-pinned-memory` (no effect on this specific crash)
- **Models storage:** local NVMe drive

## Steps to Reproduce

**Case 1 — MiniMax H3, Image-to-Video (official template)**

1. Load the official MiniMax H3 FL2VA Image-to-Video workflow with:
- `unet_name`: `minimax_h3_fl2va_pruned_int8_convrot.safetensors`
- `clip_name`: `qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors`
- `vae_name`: `minimax_h3_video_vae_fp16.safetensors`
- `audio_vae`: `minimax_h3_audio_vae_fp32.safetensors`
2. Queue a prompt.
3. Process crashes ~1–2 minutes in, consistently at ~35% of the reported task progress. No unusual VRAM/RAM growth is visible in Task Manager leading up to the crash.
**Case 2 — LTX-2, Image-to-Video**

1. Load an LTX-2 Image-to-Video workflow with:
- `ckpt_name`: `ltx-2-19b-dev-fp8.safetensors`
- `text_encoder`: `gemma_3_12B_it_fp4_mixed.safetensors`
- `lora_name`: `ltx-2-19b-distilled-lora-384.safetensors`
- `model_name`: `ltx-2-spatial-upscaler-x2-1.0.safetensors`
2. Queue a prompt.
3. Process crashes within ~20 seconds, **before any GPU/RAM utilization increase** is visible in Task Manager.

### Debug Logs

```powershell
Crash Log (captured from the LTX-2 reproduction, Case 2)

[INFO] got prompt
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float32
[INFO] Found quantization metadata version 1
[INFO] Detected mixed precision quantization
[INFO] Using mixed precision operations
[INFO] Native ops: float8_e5m2, convrot_w4a4, int8_tensorwise, float8_e4m3fn , emulated ops: mxfp8, nvfp4
[INFO] model weight dtype torch.bfloat16, manual cast: torch.bfloat16
[INFO] model_type FLUX
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[WARNING] no CLIP/text encoder weights in checkpoint, the text encoder model will not be loaded.
[INFO] Requested to load VideoVAE
[INFO] loaded completely; 14639.80 MB usable, 2331.69 MB loaded, full load: True
Windows fatal exception: access violation

Stack (most recent call first):
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\.venv\Lib\site-packages\torch\storage.py", line 470 in __getitem__
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\comfy\utils.py", line 136 in load_torch_file
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\comfy\sd.py", line 1454 in load_clip
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\comfy_extras\nodes_lt_audio.py", line 206 in execute
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\comfy_api\latest\_io.py", line 1935 in EXECUTE_NORMALIZED
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\comfy_api\internal\__init__.py", line 149 in wrapped_func
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\execution.py", line 306 in process_inputs
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\execution.py", line 318 in _async_map_node_over_list
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\execution.py", line 344 in get_output_data
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\execution.py", line 545 in execute
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\execution.py", line 789 in execute_async
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\standalone-env\Lib\asyncio\events.py", line 88 in _run
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\standalone-env\Lib\asyncio\base_events.py", line 1999 in _run_once
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\standalone-env\Lib\asyncio\base_events.py", line 645 in run_forever
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\standalone-env\Lib\asyncio\windows_events.py", line 322 in run_forever
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\standalone-env\Lib\asyncio\base_events.py", line 678 in run_until_complete
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\standalone-env\Lib\asyncio\runners.py", line 118 in run
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\standalone-env\Lib\asyncio\runners.py", line 195 in run
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\execution.py", line 728 in execute
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\ComfyUI\main.py", line 372 in prompt_worker
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\standalone-env\Lib\threading.py", line 1012 in run
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\standalone-env\Lib\threading.py", line 1075 in _bootstrap_inner
File "F:\Comfy-Desktop\ComfyUI-Installs\ComfyUI\standalone-env\Lib\threading.py", line 1032 in _bootstrap
Exception Code: 0xC0000005
0x00007FFB22983011, torch_cpu.dll + 0x8D3011 byte(s), ?_local_scalar_dense_cpu@native@at@@YA?AVScalar@c10@@AEBVTensor@2@@Z() + 0xF1 byte(s)
0x00007FFB23612542, torch_cpu.dll + 0x1562542 byte(s), ?call@_local_scalar_dense@_ops@at@@SA?AVScalar@c10@@AEBVTensor@3@@Z() + 0xD2 byte(s)
0x00007FFB22982A15, torch_cpu.dll + 0x8D2A15 byte(s), ?item@native@at@@YA?AVScalar@c10@@AEBVTensor@2@@Z() + 0x155 byte(s)
0x00007FFB23319352, torch_cpu.dll + 0x1269352 byte(s), ?call@item@_ops@at@@SA?AVScalar@c10@@AEBVTensor@3@@Z() + 0xD2 byte(s)
0x00007FFB246AFB63, torch_cpu.dll + 0x25FFB63 byte(s), ??$item@E@Tensor@at@@QEBAEXZ() + 0x23 byte(s)
0x00007FFB151A7A23, torch_python.dll + 0xBB7A23 byte(s), ??4?$THPPointer@UTHPGenerator@@@@QEAAAEAV0@$$QEAV0@@Z() + 0x633 byte(s)
0x00007FFB14B6191A, torch_python.dll + 0x57191A byte(s), ?THPStorage_assertNotNull@@YAXPEAU_object@@@Z() + 0x13BA byte(s)
0x00007FFB15217BF3, torch_python.dll + 0xC27BF3 byte(s), ?legacy_tensor_ctor@utils@torch@@YA?AVTensor@at@@W4DispatchKey@c10@@W4ScalarType@6@PEAU_object@@2@Z() + 0xFE3 byte(s)
0x00007FFB1521DC2D, torch_python.dll + 0xC2DC2D byte(s), ?legacy_tensor_ctor@utils@torch@@YA?AVTensor@at@@W4DispatchKey@c10@@W4ScalarType@6@PEAU_object@@2@Z() + 0x701D byte(s)
0x00007FFBEA0EFC51, safetensors\_safetensors_rust.pyd + 0x1FC51 byte(s)
0x00007FFBEA0E2EA7, safetensors\_safetensors_rust.pyd + 0x12EA7 byte(s)
0x00007FFBEA0EA9AA, safetensors\_safetensors_rust.pyd + 0x1A9AA byte(s)
0x00007FFBEA0F2F52, safetensors\_safetensors_rust.pyd + 0x22F52 byte(s)
0x00007FFBEA0F3354, safetensors\_safetensors_rust.pyd + 0x23354 byte(s)
[... CPython interpreter frames omitted for brevity ...]
0x00007FFC203EAD6C, ntdll.dll + 0xAAD6C byte(s), RtlUserThreadStart() + 0x2C byte(s)
```

### Other

## Observations / Suspected Root Cause

- Both crash instances occur inside `load_torch_file()` → the `safetensors` Rust extension reading a tensor from a quantized checkpoint → PyTorch tensor construction (`legacy_tensor_ctor`) → a `.item()` call inside `torch_cpu.dll` (`_local_scalar_dense_cpu`), which then raises an access violation while reading the tensor's underlying storage.
- The startup log explicitly separates: `Native ops: float8_e5m2, convrot_w4a4, int8_tensorwise, float8_e4m3fn` vs. `emulated ops: mxfp8, nvfp4`.
- **Both files that trigger the crash use NVFP4 specifically** — `qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` (MiniMax H3) and `gemma_3_12B_it_fp4_mixed.safetensors` (LTX-2) — which is listed as an *emulated* op rather than a native one.
- No abnormal VRAM/system RAM usage was observed leading up to either crash, which rules out plain out-of-memory as the cause.
- `--disable-pinned-memory` was tested and had no effect on this specific crash.
- Windows Event Viewer shows no corresponding Application Error or GPU-driver-reset (TDR) event at the time of the crash — the fault appears to be caught by the native fault handler before Windows logs it as an unhandled process crash.
- **Hypothesis:** the bug may be specific to the NVFP4 "emulated" dequantization/metadata-reading path — reading a per-block scale or zero-point value in this format may produce a scalar tensor whose backing storage is invalid or has already been freed, triggering the access violation on `.item()`.
- Not yet independently confirmed, but switching both affected text encoders to files that use native quantization ops instead (e.g. `qwen3vl_32b_minimax_h3_int8_convrot.safetensors` for MiniMax H3, `gemma_3_12B_it_fp8_scaled.safetensors` for LTX-2) is planned as a follow-up test to isolate whether NVFP4 specifically is the trigger, or whether it is the broader mixed-precision quant-metadata loader.
## Additional Context

- Model files are stored on a local NVMe drive, ruling out slow-storage I/O timing as a factor.

Contributor guide

Open the contributing guide

Research direction

Start with comfy/utils.py load_torch_file and follow the call from comfy/sd.py load_clip through comfy_extras/nodes_lt_audio.py execute. Reproduce with the NVFP4 text-encoder files, then compare against the native-quantization files named in the report while inspecting the mixed-precision metadata path. Done means incompatible input produces a handled Python error instead of a native access violation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.