Comfy-Org / Comfy-Org/ComfyUI

[ROCm/Windows][gfx1151] DynamicVRAM: access violation in vbar_allocate when loading LTXVLatentUpsampler

Open
#16,102 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists

Only `websocket_image_save.py` (built-in) and ComfyUI-Manager are installed. Neither is in the failing graph.

### Expected Behavior

The stock `LTX-2.5: Image to Video` template (`video_ltx2_5_i2v`) should complete: base sampling, latent upscale, VAE decode, save.

### Actual Behavior

Base sampling completes normally (8/8 @ ~2.9 s/it). Execution then dies at `LTXVLatentUpsampler` while allocating the model's VBAR:

```
Requested to load LatentUpsampler
!!! Exception during processing !!! exception: access violation reading 0x00000000000000E0
```

Traceback tail (paths abbreviated):

```
File "...\ComfyUI\comfy_extras\nodes_lt_upsampler.py", line 48, in execute
model_management.load_models_gpu([upscale_model], memory_required=memory_required)
File "...\ComfyUI\comfy\model_management.py", line 1004, in load_models_gpu
loaded_model.model_load(lowvram_model_memory, force_patch_weights=force_patch_weights)
File "...\ComfyUI\comfy\model_management.py", line 790, in model_load
self.model_use_more_vram(use_more_vram, force_patch_weights=force_patch_weights)
File "...\ComfyUI\comfy\model_management.py", line 818, in model_use_more_vram
return self.model.partially_load(self.device, extra_memory, force_patch_weights=force_patch_weights)
File "...\ComfyUI\comfy\model_patcher.py", line 2150, in partially_load
self.load(device_to, dirty=dirty)
File "...\ComfyUI\comfy\model_patcher.py", line 1873, in load
vbar = self._vbar_get(create=True)
File "...\ComfyUI\comfy\model_patcher.py", line 1805, in _vbar_get
vbar = comfy_aimdo.model_vbar.ModelVBAR(self.model_size() * 10, self.load_device.index)
File "...\site-packages\comfy_aimdo\model_vbar.py", line 52, in __init__
self._ptr = lib.vbar_allocate(self._devctx, int(size), device)
OSError: exception: access violation reading 0x00000000000000E0
```

Reading offset `0xE0` from a null pointer suggests `_devctx` (or a field it points at) is null inside `vbar_allocate` for this model, rather than a genuine allocation failure.

### This is not memory pressure

At the moment of the crash:

- Total VRAM 110457 MB, ~107 GB free
- `vram_free` from `/system_stats` immediately before the run: 115661697024 bytes
- The two **large** models in the same graph staged through the same DynamicVRAM path without incident:
- `Model LTXAVTEModel_ prepared for dynamic VRAM loading. 14612MB Staged.`
- `Model LTXAV prepared for dynamic VRAM loading. 20484MB Staged.`

It is the small (~950 MB) LatentUpsampler that fails. Unlike #15606 this is not size-dependent — it reproduces at 0.4 MP / 5s — and it raises a catchable `OSError` rather than killing the process.

### Workaround, and why it may matter

Launching with `--disable-dynamic-vram` fixes it completely. The identical workflow file, same seed, then runs to completion in 140.47s.

The workaround costs **no measurable performance**:

| Stage | DynamicVRAM on | DynamicVRAM off |
|---|---|---|
| Base sampling, 8 steps | 2.92 s/it (crashed after) | 2.99 s/it |
| Latent upscale, 3 steps | never reached | 10.22 s/it |
| Total | failed at 55.03s | **140.47s, completed** |

With the flag, `LTXAV` full-loads instead — `loaded completely; 92365.80 MB usable, 20487.25 MB loaded, full load: True` — and `LatentUpsampler` loads in ~190 ms at 949.61 MB.

Raising this because the startup message states the flag "will be removed soon" and asks for detailed reports. On this hardware, removing it would currently make LTX-2.5 unusable.

### Steps to Reproduce

1. Windows 11 + ROCm on a gfx1151 iGPU, DynamicVRAM enabled (the default here).
2. Fetch the stock template `video_ltx2_5_i2v`.
3. 0.4 MP, 16:9, `multiple` 32, 5s @ 24fps. Prompt enhancement off, latent upscaler enabled (default).
4. Models: `ltx-2.5-22b-distilled-transformer-comfy-int8-convrot`, `gemma4-12b-with-proj-ltx-2.5-comfy-int8-convrot`, `ltx-2.5-video-vae-bf16`, `ltx-2.5-audio-vae-bf16`, `ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0`.
5. Queue. Base sampling completes; `LTXVLatentUpsampler` raises the access violation.
6. Add `--disable-dynamic-vram`, re-run the identical file — completes.

Reproduces 100% of the time and fails in under a minute.

### Regression note

This workflow completed successfully on **ComfyUI 0.33.2 / comfy-aimdo 0.4.13**, where `DynamicVRAM support detected and enabled` did not appear in the startup log at all. It fails on **0.34.3 / 0.4.15**, where it does.

### System Info

| | |
|---|---|
| OS | Windows 11 Pro (25H2) |
| App | ComfyUI Desktop, bundled standalone env, variant `win-amd` |
| GPU | Radeon 8060S iGPU (Strix Halo APU), **gfx1151**, 40 CU |
| VRAM | 110457 MB (BIOS UMA split) |
| Python | 3.13.12 |
| torch | 2.12.0+rocm7.14.0 |
| ROCm | 7.14 |
| ComfyUI | 0.34.3 (`87465b8f`) |
| comfy-aimdo | 0.4.15 |
| comfy-kitchen | 0.2.31 |
| frontend / templates | 1.49.6 / 0.11.54 |
| launchArgs | `--enable-manager` (crash) / `--enable-manager --disable-dynamic-vram` (works) |

Startup backends:

```
comfy_kitchen backend triton: available False (No module named 'triton') -- expected on Windows ROCm
comfy_kitchen backend eager: available True
comfy_kitchen backend hip: available True
Native ops: convrot_w4a4, asym_w4a8_int8, int8_tensorwise
Emulated ops: mxfp8, float8_e4m3fn, nvfp4, float8_e5m2
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon(TM) 8060S Graphics : native
```

No `PYTORCH_CUDA_ALLOC_CONF` set. No `--highvram`, no `--reserve-vram`.

### Possibly related

- #15993 — Windows ROCm, retained `ModelVBAR` reservation after full unload
- #15606 — LTX-2.5 access violation at VAE decode (different node and stack; size-dependent, kills the process)
- #16062 — ROCm/Windows, DynamicVRAM stalling during VAE decode
- #15269 — DynamicVRAM + AIMDO 0.4.11, "VRAM Allocation failed (non OOM)"
- #12726 — DynamicVRAM CUFFT_INTERNAL_ERROR, same `--disable-dynamic-vram` workaround

Happy to run instrumented builds or capture additional traces.

Contributor guide

Open the contributing guide

Research direction

Reproduce the stock `video_ltx2_5_i2v` workflow on the stated Windows ROCm setup, then compare DynamicVRAM with `--disable-dynamic-vram`. Start with `comfy_extras/nodes_lt_upsampler.py`, `comfy/model_management.py`, `comfy/model_patcher.py`, and `comfy_aimdo/model_vbar.py`, focusing on the `ModelVBAR` allocation path. Done means the latent upscaler completes without the access violation while DynamicVRAM remains enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.