comfy-aimdo: hostbuf_read_file_slice 'device copy failed' / 'aimdo memory compile error' when two ComfyUI instances stage large models on two GPUs concurrently
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 155
Description
### Summary
Running two ComfyUI instances (one per GPU) concurrently, the instance on the smaller-VRAM card aborts while staging model weights:
```
[ERROR] aimdo: src/hostbuf.c:283:ERROR:hostbuf_read_file_slice: device copy failed result=2 device_ptr=0000000DC3C98000 device=0 stream=000001D42D619560 size=67108864
[ERROR] !!! Exception during processing !!! aimdo memory compile error
RuntimeError: HostBuffer.read_file_slice failed
RuntimeError: aimdo memory compile error
```
Each instance works **on its own**. The crash only happens when **both run at the same time**.
### Environment
- ComfyUI **0.35.0** (commit `54e03f536`), Windows 11, per-install Python venv
- **comfy-aimdo 0.5.3**
- 2 GPUs: RTX 3090 24GB (instance A, `--cuda-device 0`) + RTX 3080 20GB (instance B, `--cuda-device 1`)
- launched with `--enable-dynamic-vram` (`--use-sage-attention` on B), separate ports and separate sqlite DBs
- Model: MiniMax-H3 video DiT (~19.5 GB, pruned int8) + a LoRA + a GGUF text encoder; 8-step sampler, BlockSparseAttention, 1.0MP, 16:9
### Reproduction
1. Start instance A on GPU0, submit a job, wait until it is **sampling**.
2. Start instance B on GPU1, submit a job.
3. B fails during `Model Initializing` (weight staging) with the error above.
### What we tried (still fails)
- Starting B only **after** A is already sampling (staggered start) — fails.
- A **completely separate ComfyUI install** for B (own `main.py`, own venv, own DB/temp), sharing only the model folder via `extra_model_paths.yaml` — fails.
- B **alone** (A idle) completes normally every time.
### Observation
The failure is in weight staging: `HostBuffer.read_file_slice` → device copy. It looks like when two processes stage large weights into GPU memory at the same time, the host-buffer copy fails on the smaller (20GB) card. Single process is fine.
### Question
Is concurrent multi-instance dynamic-VRAM staging a supported scenario? Could this failure be handled more gracefully (retry / fallback to a slower path) instead of aborting the run, or is there a recommended flag/limit (e.g. serialize staging) to avoid it?
### Traceback
```
File "...\site-packages\comfy_aimdo\host_buffer.py", line 109, in read_file_slice
raise RuntimeError("HostBuffer.read_file_slice failed")
File "...\site-packages\comfy_aimdo\malloc_graph.py", line 31, in abort
raise RuntimeError("aimdo memory compile error")
```
Contributor guide
Research direction
Reproduce concurrent staging with the two ComfyUI instances and GPUs described. Start with comfy_aimdo/host_buffer.py at read_file_slice and malloc_graph.py at abort, then compare with the hostbuf.c:283 device-copy failure. Done means determining whether the scenario is supported and either identifying a documented limit or defining graceful retry/fallback behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100