Comfy-Org / Comfy-Org/ComfyUI

Regression: RTX 5090 + WAN 2.2 causes cudaErrorIllegalAddress / GPU driver crash after ComfyUI 0.3.68

Open
#14,714 1 comment 0 reactions 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

WAN 2.2 rendering without PC crash or torch.AcceleratorError:
CUDA error:
an illegal memory access was encountered
and
offload_stream.synchronize()

Hardware:
GPU:
NVIDIA RTX 5090 32GB
CPU:
Intel Core Ultra 7 265K
Motherboard:
ASUS TUF GAMING B860M
RAM:
32GB
Power Supply:
MSI MAG A1000GL PCIE5 II
Native 12V-2x6 cable
Driver version according to GPU-Z: 32.0.16.1047 (NVIDIA 610.47) DCH / WIN11 64

Working configuration on this hardware:
startup includes message "working around nvidia conv3d memory bug."
ComfyUI 0.3.68
PyTorch 2.9.0+cu130
Python 3.13.9

it behaves as expected, rendering WAN 2.2 video's without issues

### Actual Behavior

same hardware

Broken configuration
ComfyUI 0.27.0
PyTorch 2.12.0+cu130
Python 3.13.12
Startup includes:
Using async weight offloading with 2 streams
Enabled pinned memory
DynamicVRAM support detected

Result:
First WAN 2.2 render may succeed (not always)
Subsequent renders often fail
Sometimes first render fails
cudaErrorIllegalAddress
Often complete GPU black screen, only option, hard reset power off PC. Nothing logged in comfyui log, so PC crash before anything can be logged.
Other image models work fine.

Hours of AI Toolkit training (including Krea 2 LoRA training) is stable on the same machine, suggesting the GPU hardware and power delivery are healthy. The issue appears specific to recent ComfyUI/WAN execution, in the versions after ComfyUI 0.3.68.

### Steps to Reproduce

Fresh portable ComfyUI
Load WAN 2.2 workflow
Generate one or more videos
Crash occurs during or shortly after sampling

Common failure locations:

start of LOW pass
occasionally during HIGH pass
What has already been ruled out

- New PSU (ATX 3.1 / native 12V-2x6)
- GPU stable under AI Toolkit training
- Same NVIDIA driver
- Same Windows install
- Same WAN model
- Same workflow
- Same LoRAs
- Fresh ComfyUI installs
- --disable-async-offload (doens't solve)
- --disable-dynamic-vram (doesn't solve)
- --disable-smart-memory (doesn't solve)
- Simplified workflow
- No TeaCache

### Debug Logs

```powershell
torch.AcceleratorError:
CUDA error:
an illegal memory access was encountered

and

offload_stream.synchronize()
Nothing else in logs because the system crashed at that time. A Hard reset is the only option
```

### Other

I tested the different version on the same hardware, the same day, no gpu drivers updates or windows updates in between.
Driver version according to GPU-Z: 32.0.16.1047 (NVIDIA 610.47) DCH / WIN11 64

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the WAN 2.2 workflow on the reported ComfyUI and PyTorch versions, comparing the working 0.3.68 configuration with the failing newer configuration. Inspect the execution around offload_stream.synchronize(), async weight offloading, and dynamic VRAM, since these are the failure context named in the report. Done means repeated WAN 2.2 renders complete without cudaErrorIllegalAddress or a GPU driver crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.