Comfy-Org / Comfy-Org/comfy-aimdo

[ROCm multi-GPU] account_free uses the current device instead of the allocation owner

Open
#99 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
67
Forks
39
Avg merge
1d 25m
Merged PRs (30d)
10

Description

## Summary

On Windows ROCm with multiple GPUs visible, comfy-aimdo 0.4.15 can fail to account a free when a pointer was allocated on one GPU but another GPU is current when the pointer is freed.

The warning is:

`account_free: could not account free`

This is reproducible without DynamicVRAM or VBAR.

## Environment

- Windows 11
- AMD Radeon RX 9070 XT ×2
- ROCm
- comfy-aimdo 0.4.15
- PyTorch with both GPUs visible

## Minimal reproduction

I tested a small allocator probe without DynamicVRAM/VBAR.

### Case 1: 2 GPUs visible, mismatched current device

1. Allocate a normal torch tensor on GPU 0.
2. Change the current device to GPU 1.
3. Free the GPU 0 allocation while GPU 1 is current.

Result:

`account_free: could not account free`

The probe produced 3 accounting misses.

The AIMDO log reported:

`Total VRAM for VBARs: 0 MB`

so this does not require DynamicVRAM/VBAR.

### Case 2: 2 GPUs visible, current device matches allocation owner

Allocate and free on the same current device.

Result:

No `account_free` misses.

### Case 3: 1 GPU visible

Run the same allocation/free probe with only one GPU visible.

Result:

No `account_free` misses.

## Diagnosis

The failure appears to occur because the free/accounting path uses the current device's allocation table rather than locating the device that actually owns the tracked allocation.

For example:

- allocation owner: GPU 0
- current device at free time: GPU 1
- accounting searches GPU 1 state
- tracked GPU 0 allocation is not found
- `account_free: could not account free`

DynamicVRAM/VBAR-heavy workflows make this easier to trigger, but VBAR is not required.

## Impact

In my larger ComfyUI test workload with stock comfy-aimdo 0.4.15, this warning occurred 119 times.

The tested workflows still completed, so I am not claiming this accounting issue alone causes a crash. However, it leaves AIMDO's per-device allocation accounting inconsistent in multi-GPU use.

## Local fix

I have tested a local fix that looks up the actual allocation owner and performs the free/accounting operation using that owner's device/context.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two-GPU probe and tracing the account_free path that reports "could not account free". Inspect how allocation ownership is tracked and make the accounting use the owning device rather than the current device; done means the mismatched-device case records frees without warnings while the single-GPU and matching-device cases still pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, pytorch
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.