Comfy-Org / Comfy-Org/comfy-aimdo
bug & fix: ModelVBAR crashes on NVIDIA GRID / vGPU instead of falling back
- Dominant language
- C
- Stars
- 67
- Forks
- 39
- Avg merge
- 1d 25m
- Merged PRs (30d)
- 10
Description
Hey,
i ran into a MemoryError on my setup even though I have enough VRAM.
Took a while to figure out but the issue is that VBAR allocation just fails silently on virtualized GPUs (vGPU environments) and then throws a hard exception instead of falling back gracefully.
`MemoryError: VBAR allocation failed
File "execution.py", line 525, in execute
File "execution.py", line 334, in get_output_data
File "custom_nodes/rgthree-comfy/py/power_prompt_simple.py", line 39, in main
conditioning = CLIPTextEncode().encode(opt_clip, prompt)[0]
File "nodes.py", line 80, in encode
return (clip.encode_from_tokens_scheduled(tokens), )
File "comfy/sd.py", line 375, in encode_from_tokens
self.load_model(tokens)
File "comfy/model_management.py", line 819, in load_models_gpu
loaded_model.model_load(lowvram_model_memory, ...)
File "comfy/model_patcher.py", line 1495, in _vbar_get
vbar = comfy_aimdo.model_vbar.ModelVBAR(self.model_size() * 10, self.load_device.index)
File "comfy_aimdo/model_vbar.py", line 51, in __init__
raise MemoryError("VBAR allocation failed")`
My setup: GRID RTX6000-24Q (24GB vGPU), Ubuntu, CUDA 12.8
VBAR cannot allocate because the underlying low-level CUDA memory ops aren't fully supported in virtualized environments.
### What I changed (i got help from AI to find a working fix)
- `ModelVBAR.__init__` no longer raises `MemoryError` when allocation fails, instead sets `_disabled = True` and emits a `RuntimeWarning`
- All methods (`alloc`, `fault`, `prioritize`, `deprioritize`, `loaded_size`, etc.) check `_disabled` and return safe dummy values instead of crashing
- `vbar_fault()` and `vbar_unpin()` module-level functions also handle `None` alloc gracefully now
ComfyUI runs fine without VBAR on these setups anyway since the optimization doesn't really work in a virtualized context, so this just makes it degrade nicely instead of hard crashing.
Or do i miss something else?
Contributor guide
Research direction
Start with comfy_aimdo/model_vbar.py and the ModelVBAR call in comfy/model_patcher.py shown in the traceback. Trace the allocation failure through ModelVBAR methods and the vbar_fault() and vbar_unpin() helpers. Done means the NVIDIA vGPU path no longer raises MemoryError and ComfyUI continues without VBAR, while normal allocation behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, python
- Domain
- ai-infra-agents, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100