Comfy-Org / Comfy-Org/ComfyUI

[Windows ROCm multi-GPU][DynamicVRAM] retained ModelVBAR reservation after full unload causes ~38 GB private-memory plateau

Open
#15,993 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

## Summary

On Windows ROCm with 2 GPUs visible and DynamicVRAM enabled, a fully unloaded model can leave its `ModelVBAR` reservation retained in ComfyUI's `dynamic_vbars`.

The resident VBAR memory can be freed while the `ModelVBAR` reservation itself remains alive.

On my system this leaves a very large Windows `MEM_PRIVATE PAGE_READWRITE` allocation committed after generation has completed.

In a controlled A/B test:

- without releasing the retained VBAR reservation on full unload:
- post-run Private Bytes: **41.58 GB**
- largest `MEM_PRIVATE PAGE_READWRITE`: **38.438 GB**
- with the retained VBAR reservation released on full unload:
- post-run Private Bytes: **7.33 GB**
- largest `MEM_PRIVATE PAGE_READWRITE`: **4.250 GB**

Generation succeeds in both cases.

The visible issue is therefore a large persistent process-private memory allocation after the model reaches the full-unload path, rather than a generation failure.

## Environment

- Windows 11
- AMD Radeon RX 9070 XT ×2
- GPU architecture: gfx1201
- PyTorch: `2.13.0+rocm10.0.0`
- ROCm: `10.0.0`
- ComfyUI: `0.34.0`
- comfy-aimdo: stock `0.4.15`
- both GPUs visible through `--cuda-device all`

## Reproduction command

```bash
python main.py --disable-smart-memory --enable-dynamic-vram --cuda-device all --disable-pinned-memory
```

Important details:

- pinned memory is explicitly disabled with `--disable-pinned-memory`
- CK Attention is not enabled
- both GPUs are visible
- no custom `SelectModelDevice` node is required
- sampler runs on `cuda:0`

`--disable-smart-memory` is included because it reliably causes the workflow to reach the full-unload path needed for this reproduction.

## Workflow

The large post-run private-memory allocation also reproduces using the official ComfyUI Krea2 workflow structure.

I did not need a custom graph, custom nodes, `SelectModelDevice`, or CK Attention to trigger the issue.

One reproduction used:

- UNet: `krea2TurboOfficialComfy_krea2TurboFp8.safetensors`
- LoRA: `krea2_darkbrush.safetensors`
- Text Encoder: `qwen3vl_4b_fp8_scaled.safetensors`
- VAE: `qwen_image_vae.safetensors`

For controlled A/B testing of the VBAR lifetime behavior, I also used a simplified Krea2 test workflow.

This therefore does not appear to be specific to one custom workflow, although I have not tested every possible workflow configuration.

## Observed behavior

Using the official ComfyUI Krea2 workflow structure and reaching full unload:

- generation succeeds
- sampler device: `cuda:0`
- post-run Working Set: **40.32 GB**
- post-run Private Bytes: **41.58 GB**
- committed private read/write memory measured with `VirtualQueryEx`: **41.079 GB**
- largest single `MEM_PRIVATE PAGE_READWRITE` allocation: **38.438 GB**
- no fatal error occurs

The ~38.4 GB private allocation remains committed after generation finishes and the server is idle.

There is no:

- `Traceback`
- `hipError`
- `AcceleratorError`
- access violation
- device mismatch

So the problem presents as a persistent RAM/private-memory plateau rather than a generation failure.

## VBAR lifetime diagnosis

The behavior appears to be caused by the lifetime of the retained `ModelVBAR` reservation rather than simply by resident VBAR pages.

I tested this independently with a direct VBAR lifetime probe:

1. create a `ModelVBAR`
2. fault memory into it
3. call the VBAR memory-free path
4. keep the `ModelVBAR` object/reservation alive

The large Windows private-memory allocation remains committed.

When the `ModelVBAR` object/reservation itself is destroyed/released, process private memory drops dramatically.

In the standalone probe, destroying the retained reservation reduced process private memory to approximately **886 MB**.

This suggests that freeing resident VBAR pages is not sufficient to release the underlying VMM-backed Windows private-memory reservation.

ComfyUI can keep the corresponding `ModelVBAR` alive through `dynamic_vbars` after a full model unload.

## Controlled A/B isolation

I tested this separately from my other local multi-GPU fixes.

Both A and B used:

- stock `comfy-aimdo 0.4.15`
- the same launch command
- pinned memory explicitly disabled from the CLI
- no CK Attention
- no MultiGPU weakref fix
- no default-device fix
- no local pinned-memory guard
- no locally modified AIMDO
- the same cast-buffer sizing workaround described below

### A. Without the VBAR lifecycle fix

- generation succeeds
- sampler: `cuda:0`
- full unload is reached
- post-run Private Bytes: **41.58 GB**
- largest `MEM_PRIVATE PAGE_READWRITE`: **38.438 GB**
- no fatal error
- large private-memory plateau remains

### B. With only the VBAR lifecycle fix added

The only additional change releases retained DynamicVRAM `ModelVBAR` reservations when a full unload is requested.

- generation succeeds
- sampler: `cuda:0`
- full unload is reached
- VBAR reservation release is confirmed
- post-run Private Bytes: **7.33 GB**
- largest `MEM_PRIVATE PAGE_READWRITE`: **4.250 GB**
- no fatal error
- the large private-memory plateau disappears

This isolates the large post-run private-memory retention from my other multi-GPU fixes.

## Local VBAR lifecycle fix

The local ComfyUI-side fix releases retained DynamicVRAM VBAR reservations when a full unload is requested.

Conceptually:

```python
if memory_to_free >= 1e30:
self.release_dynamic_vbars()
```

`release_dynamic_vbars()` releases/destroys the retained `ModelVBAR` objects/reservations instead of only freeing their resident pages.

## Reproduction prerequisite / related cast-buffer issue

There is one additional complication on this Windows ROCm multi-GPU system.

With the otherwise stock-like ComfyUI configuration, an earlier cast-buffer/VMM failure occurs before the workflow reaches a clean full unload.

This appears related to existing Windows ROCm AIMDO reports, including:

- `Comfy-Org/comfy-aimdo#45`
- `Comfy-Org/comfy-aimdo#63`

For the controlled A/B test above, I therefore kept only a minimal cast-buffer sizing workaround enabled so that the workflow could reach the full-unload path.

Without that workaround, the test fails earlier in the VMM/cast-buffer path with errors observed during testing such as:

- `cuMemMap` / HIP mapping failure
- error 719 / launch failure
- `Fault failed: 2`

The cast-buffer workaround is identical in both A and B.

Therefore I do not believe the cast-buffer issue causes the retained-`ModelVBAR` issue. It appears to mask it on this system by failing before the workflow reaches the state where the VBAR lifetime problem can be observed.

I am treating the retained `ModelVBAR` reservation as the primary issue here, while mentioning the cast-buffer problem only because it is required to explain the reproduction conditions.

## Pinned-memory / attention isolation

The reproduction explicitly uses:

```bash
--disable-pinned-memory
```

and does not use:

```bash
--use-ck-attention
```

The ~38.438 GB private-memory allocation still reproduces.

Therefore this retained-VBAR/private-memory issue does not require CK Attention and reproduces with the pinned-memory path disabled.

## Expected behavior

When a DynamicVRAM model is fully unloaded, ComfyUI should release both:

- resident VBAR memory
- the retained `ModelVBAR` reservation

so that large VMM-backed private-memory allocations are not kept committed after the model has been fully unloaded.

Contributor guide

Open the contributing guide

Research direction

Trace ComfyUI's full-unload path and the dynamic_vbars collection, starting with how retained ModelVBAR objects are handled. Reproduce with the listed main.py command and compare the reservation lifetime with the described local release_dynamic_vbars() fix. Done means a full unload releases both resident VBAR memory and the retained reservation without the large private-memory plateau.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.