Comfy-Org / Comfy-Org/comfy-aimdo
[ROCm multi-GPU] account_free uses the current device instead of the allocation owner
- Dominant language
- C
- Stars
- 67
- Forks
- 39
- Avg merge
- 1d 25m
- Merged PRs (30d)
- 10
Description
## Summary
On Windows ROCm with multiple GPUs visible, comfy-aimdo 0.4.15 can fail to account a free when a pointer was allocated on one GPU but another GPU is current when the pointer is freed.
The warning is:
`account_free: could not account free`
This is reproducible without DynamicVRAM or VBAR.
## Environment
- Windows 11
- AMD Radeon RX 9070 XT ×2
- ROCm
- comfy-aimdo 0.4.15
- PyTorch with both GPUs visible
## Minimal reproduction
I tested a small allocator probe without DynamicVRAM/VBAR.
### Case 1: 2 GPUs visible, mismatched current device
1. Allocate a normal torch tensor on GPU 0.
2. Change the current device to GPU 1.
3. Free the GPU 0 allocation while GPU 1 is current.
Result:
`account_free: could not account free`
The probe produced 3 accounting misses.
The AIMDO log reported:
`Total VRAM for VBARs: 0 MB`
so this does not require DynamicVRAM/VBAR.
### Case 2: 2 GPUs visible, current device matches allocation owner
Allocate and free on the same current device.
Result:
No `account_free` misses.
### Case 3: 1 GPU visible
Run the same allocation/free probe with only one GPU visible.
Result:
No `account_free` misses.
## Diagnosis
The failure appears to occur because the free/accounting path uses the current device's allocation table rather than locating the device that actually owns the tracked allocation.
For example:
- allocation owner: GPU 0
- current device at free time: GPU 1
- accounting searches GPU 1 state
- tracked GPU 0 allocation is not found
- `account_free: could not account free`
DynamicVRAM/VBAR-heavy workflows make this easier to trigger, but VBAR is not required.
## Impact
In my larger ComfyUI test workload with stock comfy-aimdo 0.4.15, this warning occurred 119 times.
The tested workflows still completed, so I am not claiming this accounting issue alone causes a crash. However, it leaves AIMDO's per-device allocation accounting inconsistent in multi-GPU use.
## Local fix
I have tested a local fix that looks up the actual allocation owner and performs the free/accounting operation using that owner's device/context.
Contributor guide
Research direction
Start by reproducing the two-GPU probe and tracing the account_free path that reports "could not account free". Inspect how allocation ownership is tracked and make the accounting use the owning device rather than the current device; done means the mismatched-device case records frees without warnings while the single-GPU and matching-device cases still pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, pytorch
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100