Comfy-Org / Comfy-Org/comfy-aimdo

bug & fix: ModelVBAR crashes on NVIDIA GRID / vGPU instead of falling back

Open
#25 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
67
Forks
39
Avg merge
1d 25m
Merged PRs (30d)
10

Description

Hey,
i ran into a MemoryError on my setup even though I have enough VRAM.
Took a while to figure out but the issue is that VBAR allocation just fails silently on virtualized GPUs (vGPU environments) and then throws a hard exception instead of falling back gracefully.

`MemoryError: VBAR allocation failed
File "execution.py", line 525, in execute
File "execution.py", line 334, in get_output_data
File "custom_nodes/rgthree-comfy/py/power_prompt_simple.py", line 39, in main
conditioning = CLIPTextEncode().encode(opt_clip, prompt)[0]
File "nodes.py", line 80, in encode
return (clip.encode_from_tokens_scheduled(tokens), )
File "comfy/sd.py", line 375, in encode_from_tokens
self.load_model(tokens)
File "comfy/model_management.py", line 819, in load_models_gpu
loaded_model.model_load(lowvram_model_memory, ...)
File "comfy/model_patcher.py", line 1495, in _vbar_get
vbar = comfy_aimdo.model_vbar.ModelVBAR(self.model_size() * 10, self.load_device.index)
File "comfy_aimdo/model_vbar.py", line 51, in __init__
raise MemoryError("VBAR allocation failed")`

My setup: GRID RTX6000-24Q (24GB vGPU), Ubuntu, CUDA 12.8
VBAR cannot allocate because the underlying low-level CUDA memory ops aren't fully supported in virtualized environments.

### What I changed (i got help from AI to find a working fix)

- `ModelVBAR.__init__` no longer raises `MemoryError` when allocation fails, instead sets `_disabled = True` and emits a `RuntimeWarning`
- All methods (`alloc`, `fault`, `prioritize`, `deprioritize`, `loaded_size`, etc.) check `_disabled` and return safe dummy values instead of crashing
- `vbar_fault()` and `vbar_unpin()` module-level functions also handle `None` alloc gracefully now

ComfyUI runs fine without VBAR on these setups anyway since the optimization doesn't really work in a virtualized context, so this just makes it degrade nicely instead of hard crashing.

Or do i miss something else?

Contributor guide

Open the contributing guide

Research direction

Start with comfy_aimdo/model_vbar.py and the ModelVBAR call in comfy/model_patcher.py shown in the traceback. Trace the allocation failure through ModelVBAR methods and the vbar_fault() and vbar_unpin() helpers. Done means the NVIDIA vGPU path no longer raises MemoryError and ComfyUI continues without VBAR, while normal allocation behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
ai-infra-agents, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.