[Windows ROCm multi-GPU][DynamicVRAM] retained ModelVBAR reservation after full unload causes ~38 GB private-memory plateau
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
## Summary
On Windows ROCm with 2 GPUs visible and DynamicVRAM enabled, a fully unloaded model can leave its `ModelVBAR` reservation retained in ComfyUI's `dynamic_vbars`.
The resident VBAR memory can be freed while the `ModelVBAR` reservation itself remains alive.
On my system this leaves a very large Windows `MEM_PRIVATE PAGE_READWRITE` allocation committed after generation has completed.
In a controlled A/B test:
- without releasing the retained VBAR reservation on full unload:
- post-run Private Bytes: **41.58 GB**
- largest `MEM_PRIVATE PAGE_READWRITE`: **38.438 GB**
- with the retained VBAR reservation released on full unload:
- post-run Private Bytes: **7.33 GB**
- largest `MEM_PRIVATE PAGE_READWRITE`: **4.250 GB**
Generation succeeds in both cases.
The visible issue is therefore a large persistent process-private memory allocation after the model reaches the full-unload path, rather than a generation failure.
## Environment
- Windows 11
- AMD Radeon RX 9070 XT ×2
- GPU architecture: gfx1201
- PyTorch: `2.13.0+rocm10.0.0`
- ROCm: `10.0.0`
- ComfyUI: `0.34.0`
- comfy-aimdo: stock `0.4.15`
- both GPUs visible through `--cuda-device all`
## Reproduction command
```bash
python main.py --disable-smart-memory --enable-dynamic-vram --cuda-device all --disable-pinned-memory
```
Important details:
- pinned memory is explicitly disabled with `--disable-pinned-memory`
- CK Attention is not enabled
- both GPUs are visible
- no custom `SelectModelDevice` node is required
- sampler runs on `cuda:0`
`--disable-smart-memory` is included because it reliably causes the workflow to reach the full-unload path needed for this reproduction.
## Workflow
The large post-run private-memory allocation also reproduces using the official ComfyUI Krea2 workflow structure.
I did not need a custom graph, custom nodes, `SelectModelDevice`, or CK Attention to trigger the issue.
One reproduction used:
- UNet: `krea2TurboOfficialComfy_krea2TurboFp8.safetensors`
- LoRA: `krea2_darkbrush.safetensors`
- Text Encoder: `qwen3vl_4b_fp8_scaled.safetensors`
- VAE: `qwen_image_vae.safetensors`
For controlled A/B testing of the VBAR lifetime behavior, I also used a simplified Krea2 test workflow.
This therefore does not appear to be specific to one custom workflow, although I have not tested every possible workflow configuration.
## Observed behavior
Using the official ComfyUI Krea2 workflow structure and reaching full unload:
- generation succeeds
- sampler device: `cuda:0`
- post-run Working Set: **40.32 GB**
- post-run Private Bytes: **41.58 GB**
- committed private read/write memory measured with `VirtualQueryEx`: **41.079 GB**
- largest single `MEM_PRIVATE PAGE_READWRITE` allocation: **38.438 GB**
- no fatal error occurs
The ~38.4 GB private allocation remains committed after generation finishes and the server is idle.
There is no:
- `Traceback`
- `hipError`
- `AcceleratorError`
- access violation
- device mismatch
So the problem presents as a persistent RAM/private-memory plateau rather than a generation failure.
## VBAR lifetime diagnosis
The behavior appears to be caused by the lifetime of the retained `ModelVBAR` reservation rather than simply by resident VBAR pages.
I tested this independently with a direct VBAR lifetime probe:
1. create a `ModelVBAR`
2. fault memory into it
3. call the VBAR memory-free path
4. keep the `ModelVBAR` object/reservation alive
The large Windows private-memory allocation remains committed.
When the `ModelVBAR` object/reservation itself is destroyed/released, process private memory drops dramatically.
In the standalone probe, destroying the retained reservation reduced process private memory to approximately **886 MB**.
This suggests that freeing resident VBAR pages is not sufficient to release the underlying VMM-backed Windows private-memory reservation.
ComfyUI can keep the corresponding `ModelVBAR` alive through `dynamic_vbars` after a full model unload.
## Controlled A/B isolation
I tested this separately from my other local multi-GPU fixes.
Both A and B used:
- stock `comfy-aimdo 0.4.15`
- the same launch command
- pinned memory explicitly disabled from the CLI
- no CK Attention
- no MultiGPU weakref fix
- no default-device fix
- no local pinned-memory guard
- no locally modified AIMDO
- the same cast-buffer sizing workaround described below
### A. Without the VBAR lifecycle fix
- generation succeeds
- sampler: `cuda:0`
- full unload is reached
- post-run Private Bytes: **41.58 GB**
- largest `MEM_PRIVATE PAGE_READWRITE`: **38.438 GB**
- no fatal error
- large private-memory plateau remains
### B. With only the VBAR lifecycle fix added
The only additional change releases retained DynamicVRAM `ModelVBAR` reservations when a full unload is requested.
- generation succeeds
- sampler: `cuda:0`
- full unload is reached
- VBAR reservation release is confirmed
- post-run Private Bytes: **7.33 GB**
- largest `MEM_PRIVATE PAGE_READWRITE`: **4.250 GB**
- no fatal error
- the large private-memory plateau disappears
This isolates the large post-run private-memory retention from my other multi-GPU fixes.
## Local VBAR lifecycle fix
The local ComfyUI-side fix releases retained DynamicVRAM VBAR reservations when a full unload is requested.
Conceptually:
```python
if memory_to_free >= 1e30:
self.release_dynamic_vbars()
```
`release_dynamic_vbars()` releases/destroys the retained `ModelVBAR` objects/reservations instead of only freeing their resident pages.
## Reproduction prerequisite / related cast-buffer issue
There is one additional complication on this Windows ROCm multi-GPU system.
With the otherwise stock-like ComfyUI configuration, an earlier cast-buffer/VMM failure occurs before the workflow reaches a clean full unload.
This appears related to existing Windows ROCm AIMDO reports, including:
- `Comfy-Org/comfy-aimdo#45`
- `Comfy-Org/comfy-aimdo#63`
For the controlled A/B test above, I therefore kept only a minimal cast-buffer sizing workaround enabled so that the workflow could reach the full-unload path.
Without that workaround, the test fails earlier in the VMM/cast-buffer path with errors observed during testing such as:
- `cuMemMap` / HIP mapping failure
- error 719 / launch failure
- `Fault failed: 2`
The cast-buffer workaround is identical in both A and B.
Therefore I do not believe the cast-buffer issue causes the retained-`ModelVBAR` issue. It appears to mask it on this system by failing before the workflow reaches the state where the VBAR lifetime problem can be observed.
I am treating the retained `ModelVBAR` reservation as the primary issue here, while mentioning the cast-buffer problem only because it is required to explain the reproduction conditions.
## Pinned-memory / attention isolation
The reproduction explicitly uses:
```bash
--disable-pinned-memory
```
and does not use:
```bash
--use-ck-attention
```
The ~38.438 GB private-memory allocation still reproduces.
Therefore this retained-VBAR/private-memory issue does not require CK Attention and reproduces with the pinned-memory path disabled.
## Expected behavior
When a DynamicVRAM model is fully unloaded, ComfyUI should release both:
- resident VBAR memory
- the retained `ModelVBAR` reservation
so that large VMM-backed private-memory allocations are not kept committed after the model has been fully unloaded.
Contributor guide
Research direction
Trace ComfyUI's full-unload path and the dynamic_vbars collection, starting with how retained ModelVBAR objects are handled. Reproduce with the listed main.py command and compare the reservation lifetime with the described local release_dynamic_vbars() fix. Done means a full unload releases both resident VBAR memory and the retained reservation without the large private-memory plateau.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100