Regression: RTX 5090 + WAN 2.2 causes cudaErrorIllegalAddress / GPU driver crash after ComfyUI 0.3.68
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
### Expected Behavior
WAN 2.2 rendering without PC crash or torch.AcceleratorError:
CUDA error:
an illegal memory access was encountered
and
offload_stream.synchronize()
Hardware:
GPU:
NVIDIA RTX 5090 32GB
CPU:
Intel Core Ultra 7 265K
Motherboard:
ASUS TUF GAMING B860M
RAM:
32GB
Power Supply:
MSI MAG A1000GL PCIE5 II
Native 12V-2x6 cable
Driver version according to GPU-Z: 32.0.16.1047 (NVIDIA 610.47) DCH / WIN11 64
Working configuration on this hardware:
startup includes message "working around nvidia conv3d memory bug."
ComfyUI 0.3.68
PyTorch 2.9.0+cu130
Python 3.13.9
it behaves as expected, rendering WAN 2.2 video's without issues
### Actual Behavior
same hardware
Broken configuration
ComfyUI 0.27.0
PyTorch 2.12.0+cu130
Python 3.13.12
Startup includes:
Using async weight offloading with 2 streams
Enabled pinned memory
DynamicVRAM support detected
Result:
First WAN 2.2 render may succeed (not always)
Subsequent renders often fail
Sometimes first render fails
cudaErrorIllegalAddress
Often complete GPU black screen, only option, hard reset power off PC. Nothing logged in comfyui log, so PC crash before anything can be logged.
Other image models work fine.
Hours of AI Toolkit training (including Krea 2 LoRA training) is stable on the same machine, suggesting the GPU hardware and power delivery are healthy. The issue appears specific to recent ComfyUI/WAN execution, in the versions after ComfyUI 0.3.68.
### Steps to Reproduce
Fresh portable ComfyUI
Load WAN 2.2 workflow
Generate one or more videos
Crash occurs during or shortly after sampling
Common failure locations:
start of LOW pass
occasionally during HIGH pass
What has already been ruled out
- New PSU (ATX 3.1 / native 12V-2x6)
- GPU stable under AI Toolkit training
- Same NVIDIA driver
- Same Windows install
- Same WAN model
- Same workflow
- Same LoRAs
- Fresh ComfyUI installs
- --disable-async-offload (doens't solve)
- --disable-dynamic-vram (doesn't solve)
- --disable-smart-memory (doesn't solve)
- Simplified workflow
- No TeaCache
### Debug Logs
```powershell
torch.AcceleratorError:
CUDA error:
an illegal memory access was encountered
and
offload_stream.synchronize()
Nothing else in logs because the system crashed at that time. A Hard reset is the only option
```
### Other
I tested the different version on the same hardware, the same day, no gpu drivers updates or windows updates in between.
Driver version according to GPU-Z: 32.0.16.1047 (NVIDIA 610.47) DCH / WIN11 64
Contributor guide
Research direction
Start by reproducing the WAN 2.2 workflow on the reported ComfyUI and PyTorch versions, comparing the working 0.3.68 configuration with the failing newer configuration. Inspect the execution around offload_stream.synchronize(), async weight offloading, and dynamic VRAM, since these are the failure context named in the report. Done means repeated WAN 2.2 renders complete without cudaErrorIllegalAddress or a GPU driver crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100