Comfy-Org / Comfy-Org/comfy-aimdo

[ROCM/Linux]: VBAR allocation fails on RDNA1

Open
#95 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
67
Forks
39
Avg merge
1d 25m
Merged PRs (30d)
10

Description

Stacktrace with a clean ComfyUI install:
```
uv run main.py[INFO] setup plugin alembic.autogenerate.schemas
[INFO] setup plugin alembic.autogenerate.tables
[INFO] setup plugin alembic.autogenerate.types
[INFO] setup plugin alembic.autogenerate.constraints
[INFO] setup plugin alembic.autogenerate.defaults
[INFO] setup plugin alembic.autogenerate.comments
[INFO] setup plugin alembic.autogenerate.checkconstraint_byname
[INFO] Found comfy_kitchen backend hip: {'available': False, 'disabled': False, 'unavailable_reason': 'architecture is not in the validated target manifest: gfx1010, gfx90c', 'capabilities': []}
[INFO] Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'int8_linear', 'na3d', 'quantize_and_rotate_rowwise', 'quantize_int8_rowwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'w4a8_int8_linear']}
[INFO] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_embedding', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'dequantize_w4a8_int8_weight', 'gemv_awq_w4a16', 'int8_linear', 'na3d', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'quantize_w4a8_int8_weight', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'rotate_int8_convrot_weight', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8', 'w4a8_int8_linear']}
[INFO] Found comfy_kitchen backend cuda: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'dequantize_w4a8_int8_weight', 'gemv_awq_w4a16', 'na3d', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'quantize_w4a8_int8_weight', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'rotate_int8_convrot_weight', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8', 'w4a8_int8_linear']}
[INFO] Checkpoint files will always be loaded safely.
[INFO] Total VRAM 6128 MB, total RAM 63664 MB
[INFO] pytorch version: 2.12.0+rocm10.1.0a20260824
[INFO] AMD arch: gfx1010
[INFO] ROCm version: (7, 16)
[INFO] Set vram state to: NORMAL_VRAM
[INFO] Device: cuda:0 AMD Radeon RX 5600M : native
[INFO] Device: cuda:1 AMD Radeon Graphics : native
[INFO] Using async weight offloading with 2 streams
[INFO] Enabled pinned memory 57297.0
[INFO] Using sub quadratic optimization for attention, if you have memory or speed issues try using: --use-split-cross-attention
[INFO] aimdo: /project/src/control.c:276:INFO:comfy-aimdo inited for GPU: AMD Radeon RX 5600M (VRAM: 6128 MB)
[INFO] aimdo: /project/src/control.c:276:INFO:comfy-aimdo inited for GPU: AMD Radeon Graphics (VRAM: 32768 MB)
[INFO] DynamicVRAM support detected and enabled
[INFO] Python version: 3.14.5 (main, May 10 2026, 19:28:16) [Clang 22.1.3 ]
[INFO] ComfyUI version: 0.33.0
[INFO] comfy-aimdo version: 0.4.13
[INFO] comfy-kitchen version: 0.2.31
[INFO] comfyui-frontend-package version: 1.49.6
[INFO] comfyui-workflow-templates version: 0.11.46
[INFO] comfyui-embedded-docs version: 0.5.10
[INFO] comfy-kitchen version: 0.2.31
[INFO] comfy-aimdo version: 0.4.13
[INFO] [Prompt Server] web root: /home/aaron/Projects/personal/clean_comfy/ComfyUI/.venv/lib/python3.14/site-packages/comfyui_frontend_package/static
[INFO] Asset seeder disabled
[INFO] No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
[INFO]
Import times for custom nodes:
[INFO] 0.0 seconds: /home/aaron/Projects/personal/clean_comfy/ComfyUI/custom_nodes/websocket_image_save.py
[INFO]
[INFO] Context impl SQLiteImpl.
[INFO] Will assume non-transactional DDL.
[INFO] Using RAM pressure cache.
[INFO] Starting server

[INFO] To see the GUI go to: http://127.0.0.1:8188
[INFO] got prompt
[INFO] model weight dtype torch.float16, manual cast: None
[INFO] model_type EPS
[INFO] Using split attention in VAE
[INFO] Using split attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float32
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] Requested to load SD1ClipModel
[ERROR] aimdo: /project/src/model-vbar.c:236:ERROR:Could not reseve Virtual Address space for VBAR
[ERROR] !!! Exception during processing !!! VBAR allocation failed
[ERROR] Traceback (most recent call last):
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/execution.py", line 545, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/execution.py", line 344, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/execution.py", line 318, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/execution.py", line 306, in process_inputs
result = f(**inputs)
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/nodes.py", line 77, in encode
return (clip.encode_from_tokens_scheduled(tokens), )
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/comfy/sd.py", line 341, in encode_from_tokens_scheduled
pooled_dict = self.encode_from_tokens(tokens, return_pooled=return_pooled, return_dict=True)
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/comfy/sd.py", line 405, in encode_from_tokens
self.load_model(tokens)
~~~~~~~~~~~~~~~^^^^^^^^
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/comfy/sd.py", line 463, in load_model
model_management.load_models_gpu([self.patcher], memory_required=memory_used)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/comfy/model_management.py", line 1004, in load_models_gpu
loaded_model.model_load(lowvram_model_memory, force_patch_weights=force_patch_weights)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/comfy/model_management.py", line 790, in model_load
self.model_use_more_vram(use_more_vram, force_patch_weights=force_patch_weights)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/comfy/model_management.py", line 818, in model_use_more_vram
return self.model.partially_load(self.device, extra_memory, force_patch_weights=force_patch_weights)
~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/comfy/model_patcher.py", line 2153, in partially_load
raise e
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/comfy/model_patcher.py", line 2150, in partially_load
self.load(device_to, dirty=dirty)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/comfy/model_patcher.py", line 1873, in load
vbar = self._vbar_get(create=True)
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/comfy/model_patcher.py", line 1805, in _vbar_get
vbar = comfy_aimdo.model_vbar.ModelVBAR(self.model_size() * 10, self.load_device.index)
File "/home/aaron/Projects/personal/clean_comfy/ComfyUI/.venv/lib/python3.14/site-packages/comfy_aimdo/model_vbar.py", line 54, in __init__
raise MemoryError("VBAR allocation failed")
MemoryError: VBAR allocation failed

[INFO] Prompt executed in 0.22 seconds

```
Enabling cudamallocasync has no effect

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure on the clean ComfyUI setup described for an RDNA1 gfx1010 device. Start with /project/src/model-vbar.c:236, then inspect comfy_aimdo/model_vbar.py:54 and ComfyUI/comfy/model_patcher.py:1805; done means VBAR allocation succeeds and the reported prompt proceeds without the MemoryError.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux, python
Domain
backend, operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.