[ROCm/Windows][gfx1151] DynamicVRAM: access violation in vbar_allocate when loading LTXVLatentUpsampler
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [x] I have tried disabling custom nodes and the issue persists
Only `websocket_image_save.py` (built-in) and ComfyUI-Manager are installed. Neither is in the failing graph.
### Expected Behavior
The stock `LTX-2.5: Image to Video` template (`video_ltx2_5_i2v`) should complete: base sampling, latent upscale, VAE decode, save.
### Actual Behavior
Base sampling completes normally (8/8 @ ~2.9 s/it). Execution then dies at `LTXVLatentUpsampler` while allocating the model's VBAR:
```
Requested to load LatentUpsampler
!!! Exception during processing !!! exception: access violation reading 0x00000000000000E0
```
Traceback tail (paths abbreviated):
```
File "...\ComfyUI\comfy_extras\nodes_lt_upsampler.py", line 48, in execute
model_management.load_models_gpu([upscale_model], memory_required=memory_required)
File "...\ComfyUI\comfy\model_management.py", line 1004, in load_models_gpu
loaded_model.model_load(lowvram_model_memory, force_patch_weights=force_patch_weights)
File "...\ComfyUI\comfy\model_management.py", line 790, in model_load
self.model_use_more_vram(use_more_vram, force_patch_weights=force_patch_weights)
File "...\ComfyUI\comfy\model_management.py", line 818, in model_use_more_vram
return self.model.partially_load(self.device, extra_memory, force_patch_weights=force_patch_weights)
File "...\ComfyUI\comfy\model_patcher.py", line 2150, in partially_load
self.load(device_to, dirty=dirty)
File "...\ComfyUI\comfy\model_patcher.py", line 1873, in load
vbar = self._vbar_get(create=True)
File "...\ComfyUI\comfy\model_patcher.py", line 1805, in _vbar_get
vbar = comfy_aimdo.model_vbar.ModelVBAR(self.model_size() * 10, self.load_device.index)
File "...\site-packages\comfy_aimdo\model_vbar.py", line 52, in __init__
self._ptr = lib.vbar_allocate(self._devctx, int(size), device)
OSError: exception: access violation reading 0x00000000000000E0
```
Reading offset `0xE0` from a null pointer suggests `_devctx` (or a field it points at) is null inside `vbar_allocate` for this model, rather than a genuine allocation failure.
### This is not memory pressure
At the moment of the crash:
- Total VRAM 110457 MB, ~107 GB free
- `vram_free` from `/system_stats` immediately before the run: 115661697024 bytes
- The two **large** models in the same graph staged through the same DynamicVRAM path without incident:
- `Model LTXAVTEModel_ prepared for dynamic VRAM loading. 14612MB Staged.`
- `Model LTXAV prepared for dynamic VRAM loading. 20484MB Staged.`
It is the small (~950 MB) LatentUpsampler that fails. Unlike #15606 this is not size-dependent — it reproduces at 0.4 MP / 5s — and it raises a catchable `OSError` rather than killing the process.
### Workaround, and why it may matter
Launching with `--disable-dynamic-vram` fixes it completely. The identical workflow file, same seed, then runs to completion in 140.47s.
The workaround costs **no measurable performance**:
| Stage | DynamicVRAM on | DynamicVRAM off |
|---|---|---|
| Base sampling, 8 steps | 2.92 s/it (crashed after) | 2.99 s/it |
| Latent upscale, 3 steps | never reached | 10.22 s/it |
| Total | failed at 55.03s | **140.47s, completed** |
With the flag, `LTXAV` full-loads instead — `loaded completely; 92365.80 MB usable, 20487.25 MB loaded, full load: True` — and `LatentUpsampler` loads in ~190 ms at 949.61 MB.
Raising this because the startup message states the flag "will be removed soon" and asks for detailed reports. On this hardware, removing it would currently make LTX-2.5 unusable.
### Steps to Reproduce
1. Windows 11 + ROCm on a gfx1151 iGPU, DynamicVRAM enabled (the default here).
2. Fetch the stock template `video_ltx2_5_i2v`.
3. 0.4 MP, 16:9, `multiple` 32, 5s @ 24fps. Prompt enhancement off, latent upscaler enabled (default).
4. Models: `ltx-2.5-22b-distilled-transformer-comfy-int8-convrot`, `gemma4-12b-with-proj-ltx-2.5-comfy-int8-convrot`, `ltx-2.5-video-vae-bf16`, `ltx-2.5-audio-vae-bf16`, `ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0`.
5. Queue. Base sampling completes; `LTXVLatentUpsampler` raises the access violation.
6. Add `--disable-dynamic-vram`, re-run the identical file — completes.
Reproduces 100% of the time and fails in under a minute.
### Regression note
This workflow completed successfully on **ComfyUI 0.33.2 / comfy-aimdo 0.4.13**, where `DynamicVRAM support detected and enabled` did not appear in the startup log at all. It fails on **0.34.3 / 0.4.15**, where it does.
### System Info
| | |
|---|---|
| OS | Windows 11 Pro (25H2) |
| App | ComfyUI Desktop, bundled standalone env, variant `win-amd` |
| GPU | Radeon 8060S iGPU (Strix Halo APU), **gfx1151**, 40 CU |
| VRAM | 110457 MB (BIOS UMA split) |
| Python | 3.13.12 |
| torch | 2.12.0+rocm7.14.0 |
| ROCm | 7.14 |
| ComfyUI | 0.34.3 (`87465b8f`) |
| comfy-aimdo | 0.4.15 |
| comfy-kitchen | 0.2.31 |
| frontend / templates | 1.49.6 / 0.11.54 |
| launchArgs | `--enable-manager` (crash) / `--enable-manager --disable-dynamic-vram` (works) |
Startup backends:
```
comfy_kitchen backend triton: available False (No module named 'triton') -- expected on Windows ROCm
comfy_kitchen backend eager: available True
comfy_kitchen backend hip: available True
Native ops: convrot_w4a4, asym_w4a8_int8, int8_tensorwise
Emulated ops: mxfp8, float8_e4m3fn, nvfp4, float8_e5m2
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon(TM) 8060S Graphics : native
```
No `PYTORCH_CUDA_ALLOC_CONF` set. No `--highvram`, no `--reserve-vram`.
### Possibly related
- #15993 — Windows ROCm, retained `ModelVBAR` reservation after full unload
- #15606 — LTX-2.5 access violation at VAE decode (different node and stack; size-dependent, kills the process)
- #16062 — ROCm/Windows, DynamicVRAM stalling during VAE decode
- #15269 — DynamicVRAM + AIMDO 0.4.11, "VRAM Allocation failed (non OOM)"
- #12726 — DynamicVRAM CUFFT_INTERNAL_ERROR, same `--disable-dynamic-vram` workaround
Happy to run instrumented builds or capture additional traces.
Contributor guide
Research direction
Reproduce the stock `video_ltx2_5_i2v` workflow on the stated Windows ROCm setup, then compare DynamicVRAM with `--disable-dynamic-vram`. Start with `comfy_extras/nodes_lt_upsampler.py`, `comfy/model_management.py`, `comfy/model_patcher.py`, and `comfy_aimdo/model_vbar.py`, focusing on the `ModelVBAR` allocation path. Done means the latent upscaler completes without the access violation while DynamicVRAM remains enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100