Comfy-Org / Comfy-Org/comfy-aimdo

Access violation reading 0x00000000000000E0 - RX 9070 XT (gfx120X|RDNA4) ROCm 7.13 on Windows

Open
#63 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
67
Forks
39
Avg merge
1d 25m
Merged PRs (30d)
10

Description

Hello everyone,

I noticied some instability problems when using Dynamic VRAM on ComfyUI. Sometimes everything works fine, but most of time, even a rerun of a workflow causes the following error.

Environment
Python: 3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
ComfyUI: 0.24.0
rocmIndex: gfx120X-all

Packages:
rocmIndexUrl https://rocm.nightlies.amd.com/v2/gfx120X-all/
torchvision 0.27.0a0+rocm7.13.0a20260416
torch 2.13.0a0+rocm7.13.0a20260416
torchaudio 2.11.0+rocm7.13.0a20260416
comfy-aimdo 0.4.9
comfy-kitchen 0.2.10

```
[INFO] got prompt
[INFO] model weight dtype torch.float16, manual cast: None
[INFO] model_type EPS
[INFO] Using split attention in VAE
[INFO] Using split attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] Requested to load SDXLClipModel
[INFO] Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached. Force pre-loaded 180 weights: 800 KB.
Windows fatal exception: access violation

Stack (most recent call first):
File "C:\Platform\ai\comfyui\environments\rocm-preview-py312\Lib\site-packages\comfy_aimdo\vram_buffer.py", line 32 in __init__
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\model_management.py", line 1346 in get_aimdo_cast_buffer
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\ops.py", line 123 in get_cast_buffer
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\ops.py", line 164 in cast_modules_with_vbar
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\ops.py", line 321 in cast_bias_weight
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\ops.py", line 752 in forward_comfy_cast_weights
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\ops.py", line 761 in forward
File "C:\Platform\ai\comfyui\environments\rocm-preview-py312\Lib\site-packages\torch\nn\modules\module.py", line 1789 in _call_impl
File "C:\Platform\ai\comfyui\environments\rocm-preview-py312\Lib\site-packages\torch\nn\modules\module.py", line 1778 in _wrapped_call_impl
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd1_clip.py", line 213 in process_tokens
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd1_clip.py", line 266 in forward
File "C:\Platform\ai\comfyui\environments\rocm-preview-py312\Lib\site-packages\torch\nn\modules\module.py", line 1789 in _call_impl
File "C:\Platform\ai\comfyui\environments\rocm-preview-py312\Lib\site-packages\torch\nn\modules\module.py", line 1778 in _wrapped_call_impl
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd1_clip.py", line 306 in encode
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd1_clip.py", line 45 in encode_token_weights
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sdxl_clip.py", line 59 in encode_token_weights
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd.py", line 395 in encode_from_tokens
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd.py", line 326 in encode_from_tokens_scheduled
File "C:\Platform\ai\comfyui\instances\rocm-preview\nodes.py", line 79 in encode
File "C:\Platform\ai\comfyui\instances\rocm-preview\execution.py", line 303 in process_inputs
File "C:\Platform\ai\comfyui\instances\rocm-preview\execution.py", line 315 in _async_map_node_over_list
File "C:\Platform\ai\comfyui\instances\rocm-preview\execution.py", line 341 in get_output_data
File "C:\Platform\ai\comfyui\instances\rocm-preview\execution.py", line 542 in execute
File "C:\Platform\ai\comfyui\instances\rocm-preview\execution.py", line 784 in execute_async
File "asyncio\events.py", line 88 in _run
File "asyncio\base_events.py", line 1999 in _run_once
File "asyncio\base_events.py", line 645 in run_forever
File "asyncio\windows_events.py", line 322 in run_forever
File "asyncio\base_events.py", line 678 in run_until_complete
File "asyncio\runners.py", line 118 in run
File "asyncio\runners.py", line 195 in run
File "C:\Platform\ai\comfyui\instances\rocm-preview\execution.py", line 724 in execute
File "C:\Platform\ai\comfyui\instances\rocm-preview\main.py", line 340 in prompt_worker
File "threading.py", line 1012 in run
File "threading.py", line 1075 in _bootstrap_inner
File "threading.py", line 1032 in _bootstrap
[ERROR] !!! Exception during processing !!! exception: access violation reading 0x00000000000000E0
[ERROR] Traceback (most recent call last):
File "C:\Platform\ai\comfyui\instances\rocm-preview\execution.py", line 542, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\execution.py", line 341, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\execution.py", line 315, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "C:\Platform\ai\comfyui\instances\rocm-preview\execution.py", line 303, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\nodes.py", line 79, in encode
return (clip.encode_from_tokens_scheduled(tokens), )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd.py", line 326, in encode_from_tokens_scheduled
pooled_dict = self.encode_from_tokens(tokens, return_pooled=return_pooled, return_dict=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd.py", line 395, in encode_from_tokens
o = self.cond_stage_model.encode_token_weights(tokens)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sdxl_clip.py", line 59, in encode_token_weights
g_out, g_pooled = self.clip_g.encode_token_weights(token_weight_pairs_g)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd1_clip.py", line 45, in encode_token_weights
o = self.encode(to_encode)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd1_clip.py", line 306, in encode
return self(tokens)
^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\environments\rocm-preview-py312\Lib\site-packages\torch\nn\modules\module.py", line 1778, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\environments\rocm-preview-py312\Lib\site-packages\torch\nn\modules\module.py", line 1789, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd1_clip.py", line 266, in forward
embeds, attention_mask, num_tokens, embeds_info = self.process_tokens(tokens, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\sd1_clip.py", line 213, in process_tokens
tokens_embed = self.transformer.get_input_embeddings()(tokens_embed, out_dtype=torch.float32)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\environments\rocm-preview-py312\Lib\site-packages\torch\nn\modules\module.py", line 1778, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\environments\rocm-preview-py312\Lib\site-packages\torch\nn\modules\module.py", line 1789, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\ops.py", line 761, in forward
return self.forward_comfy_cast_weights(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\ops.py", line 752, in forward_comfy_cast_weights
weight, bias, offload_stream = cast_bias_weight(self, device=input.device, dtype=out_dtype, offloadable=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\ops.py", line 321, in cast_bias_weight
offload_stream = cast_modules_with_vbar([s], dtype, device, bias_dtype, non_blocking)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\ops.py", line 164, in cast_modules_with_vbar
xfer_dest = get_cast_buffer(dest_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\ops.py", line 123, in get_cast_buffer
cast_buffer = comfy.model_management.get_aimdo_cast_buffer(offload_stream, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\instances\rocm-preview\comfy\model_management.py", line 1346, in get_aimdo_cast_buffer
cast_buffer = comfy_aimdo.vram_buffer.VRAMBuffer(DEFAULT_AIMDO_CAST_BUFFER_RESERVATION_SIZE, device.index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Platform\ai\comfyui\environments\rocm-preview-py312\Lib\site-packages\comfy_aimdo\vram_buffer.py", line 32, in __init__
self._ptr = lib.vrambuf_create(self.device, self.max_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: exception: access violation reading 0x00000000000000E0

[INFO] Prompt executed in 0.83 seconds
```

I've tried different combinations of parameters and models, and problem persists until I completely disable Dynamic VRAM, and as the --disable-dynamic-vram is announced to be removed in the future, I want to provide some information.

Thank you!

Complete log: [comfyui_dynamic_vram_rocm_windows.txt](https://github.com/user-attachments/files/28959783/comfyui_dynamic_vram_rocm_windows.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.