DynamicVRAM + AIMDO 0.4.11 causes “VRAM Allocation failed (non OOM)” and “Fault failed: 2” on second generation (QwenImage)
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [ ] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
### Expected Behavior
DynamicVRAM should correctly stream model weights across multiple generations without leaving stale VRAM segments or staged patches between runs. The second generation in queue should initialize the model normally, just like the first one, without triggering AIMDO errors such as “VRAM Allocation failed (non OOM)” or “Fault failed: 2”. Queue execution should work reliably without requiring manual intervention (pressing Run again). AIMDO should be able to reallocate VRAM segments safely on every run.
### Actual Behavior
The first generation always works correctly. The problem appears only on the second generation in queue. As soon as the second run starts, the model initialization fails immediately with the following errors:
aimdo: VRAM Allocation failed (non OOM)
RuntimeError: Fault failed: 2
The failure happens before sampling begins, during DynamicVRAM staging of QwenImage (Qwen AIO). The log shows that the model is staged (19483MB, 720 patches), but DynamicVRAM cannot reallocate or reuse the staged patches on the second run.
If I press Run manually after the failure, the generation works normally again. So the issue only affects automatic queue execution. Manual runs always succeed.
Disabling DynamicVRAM completely fixes the issue. With DynamicVRAM disabled, both the first and second generation work without errors.
For now, the issue seems do not occur on other GPUs I tested (RTX 3050 8GB and RTX 3060 12GB). It only happens on the RTX 3050 Ti 4GB laptop GPU (i have 64gb system ram ddr4)
### Steps to Reproduce
1. Start ComfyUI 0.30.1 normally (no special flags except --lowvram).
2. Make sure DynamicVRAM is enabled (default behavior).
3. Load the QwenImage (Qwen AIO) model.
4. Run any workflow that uses QwenImage. The first generation will complete successfully.
5. Queue a second generation immediately after the first one finishes.
6. As soon as the second run starts, ComfyUI fails during model initialization with:
- “VRAM Allocation failed (non OOM)”
- “RuntimeError: Fault failed: 2”
7. Press Run manually after the failure: the generation works again without errors.
8. Repeat: every second queued run fails, every manual run works.
9. Disable DynamicVRAM (using --disable-dynamic-vram): both first and second runs work normally.
### Debug Logs
```powershell
Below are the full logs from ComfyUI, including startup logs and the complete error output from the failing second generation.
Startup log:
(Everything from ComfyUI launch until “To see the GUI go to:”)
[INFO] Model QwenImage prepared for dynamic VRAM loading. 19483MB Staged. 720 patches attached. Force pre-loaded 241 weights: 72 KB.
Error during second generation:
aimdo: src/model-vbar.c:393:ERROR:VRAM Allocation failed (non OOM)
RuntimeError: Fault failed: 2
Full traceback:
[ERROR] !!! Exception during processing !!! Fault failed: 2
Traceback (most recent call last):
File "execution.py", line 545, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(...)
File "execution.py", line 344, in get_output_data
return_values = await _async_map_node_over_list(...)
File "execution.py", line 318, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "execution.py", line 306, in process_inputs
result = f(**inputs)
File "nodes.py", line 1607, in sample
return common_ksampler(...)
File "nodes.py", line 1571, in common_ksampler
samples = comfy.sample.sample(...)
File "comfy/sample.py", line 74, in sample
samples = sampler.sample(...)
File "comfy/samplers.py", line 1461, in sample
return sample(...)
File "comfy/samplers.py", line 1351, in sample
return cfg_guider.sample(...)
File "comfy/samplers.py", line 1333, in sample
output = executor.execute(...)
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "comfy/samplers.py", line 1260, in outer_sample
output = self.inner_sample(...)
File "comfy/samplers.py", line 1235, in inner_sample
samples = executor.execute(...)
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "comfy/samplers.py", line 1005, in sample
samples = self.sampler_function(...)
File "torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
File "comfy/k_diffusion/sampling.py", line 218, in sample_euler_ancestral
return sample_euler_ancestral_RF(...)
File "torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
File "comfy/k_diffusion/sampling.py", line 248, in sample_euler_ancestral_RF
denoised = model(...)
File "comfy/samplers.py", line 640, in __call__
out = self.inner_model(...)
File "comfy/samplers.py", line 1208, in __call__
return self.outer_predict_noise(...)
File "comfy/samplers.py", line 1215, in outer_predict_noise
).execute(...)
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "comfy/samplers.py", line 1218, in predict_noise
return sampling_function(...)
File "comfy/samplers.py", line 620, in sampling_function
out = calc_cond_batch(...)
File "comfy/samplers.py", line 211, in calc_cond_batch
return _calc_cond_batch_outer(...)
File "comfy/samplers.py", line 219, in _calc_cond_batch_outer
return executor.execute(...)
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "comfy/samplers.py", line 335, in _calc_cond_batch
output = model.apply_model(...).chunk(batch_chunks)
File "comfy/model_base.py", line 196, in apply_model
return comfy.patcher_extension.WrapperExecutor.new_class_executor(...)
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "comfy/model_base.py", line 240, in _apply_model
model_output = self.diffusion_model(...)
File "torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "comfy/ldm/qwen_image/model.py", line 430, in forward
return comfy.patcher_extension.WrapperExecutor.new_class_executor(...)
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "comfy/ldm/qwen_image/model.py", line 537, in _forward
encoder_hidden_states, hidden_states = block(...)
File "torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "comfy/ldm/qwen_image/model.py", line 284, in forward
txt_mod_params = self.txt_mod(temb)
File "torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "comfy/ops.py", line 547, in forward
return self.forward_comfy_cast_weights(...)
File "comfy/ops.py", line 539, in forward_comfy_cast_weights
weight, bias, offload_stream = cast_bias_weight(...)
File "comfy/ops.py", line 369, in cast_bias_weight
offload_stream = cast_modules_with_vbar(...)
File "comfy/ops.py", line 162, in cast_modules_with_vbar
signature = comfy_aimdo.model_vbar.vbar_fault(s._v)
File "comfy_aimdo/model_vbar.py", line 137, in vbar_fault
return vbar.fault(offset, size)
File "comfy_aimdo/model_vbar.py", line 94, in fault
raise RuntimeError(f"Fault failed: {res}")
End of logs.
```
### Other
I tested the same workflow on two other GPUs (RTX 3050 8GB and RTX 3060 12GB) and the issue does not occur for now. This suggests the problem is specifically related to DynamicVRAM behavior on low‑VRAM GPUs (4GB) combined with QwenImage and AIMDO 0.4.11.
Contributor guide
Assessment
This issue has not been assessed yet.