AIMDO host buffer grows until exhausted during a long single prompt: 618x hostbuf_grow ERROR, ~12% slowdown (Krea2 + LoRA, DynamicVRAM)
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Expected Behavior
Running the same Krea2 (FLUX, mixed-precision quantized, 256 LoRA patches) sampler repeatedly inside a single queued prompt should keep reusing the same pinned host buffer. Steady-state pinned host memory should not grow with the iteration count.
### Actual Behavior
The AIMDO host buffer grows monotonically across sampler iterations until it hits its reservation cap, then **every** subsequent grow fails:
```
aimdo: src/hostbuf.c:46:ERROR:hostbuf_grow: requested 27002896384 bytes beyond reserved host buffer 26998145024
```
Details from one run:
* Reservation cap: `26998145024` bytes (~25.14 GiB) = exactly `2 x model_size` for the 12864 MB staged Krea2 model, i.e. `pinned_hostbuf_size()`.
* Iterations 1..17 of the loop run clean. From iteration 18 (`15:38:49`) onward the error fires continuously — **618 times** in ~45 seconds of wall clock.
* Requested sizes creep up in small steps (1.3 MB - 11 MB) across a narrow band `26998939648 .. 27010842624`, i.e. the buffer is pinned right at the cap and each new small allocation is refused.
* Nothing is freed / compacted in between - once it is full it stays full for the rest of the prompt.
It is **not fatal** - generation continues and images still come out - but:
* Throughput degrades: `1.30s/it -> 1.38s/it` before, `1.52s/it -> 1.56s/it` after the buffer is exhausted (~12% slower).
* The log gets flooded with 618 identical ERROR lines, which makes any other problem in the same session impossible to read.
* There is no Python-side warning at all, so from the UI it just looks like ComfyUI got slower for no reason.
Total system RAM is 97918 MB, `Enabled pinned memory 39167.0` (MB), so this is not the machine running out of RAM - the per-model reservation is what is exhausted.
### Steps to Reproduce
1. Windows 11, RTX 4090 Laptop (16 GB VRAM), 96 GB RAM - DynamicVRAM / AIMDO auto-enabled.
2. Load a large mixed-precision quantized FLUX checkpoint (Krea2 here, 12864 MB staged) with LoRAs attached (256 patches).
3. Queue a **single** prompt whose sampler runs many times (batch / repeat loop) - 21 sampler runs of 8 steps in my case.
4. Watch the log. Around iteration ~18 the `hostbuf_grow` errors start and never stop until the prompt ends.
Workflow is a plain txt2img Krea2 graph; no custom node is involved in the sampling path (custom nodes present but the failing allocation is in `comfy/model_patcher.py` -> `comfy_aimdo.host_buffer`).
Launch args: `main.py --windows-standalone-build --use-sage-attention --listen --enable-cors-header`
### Debug Logs
```powershell
## ComfyUI-Manager: installing dependencies done.
[2026-08-13 15:30:49.084] ** ComfyUI startup time: 2026-08-13 15:30:49.084
[2026-08-13 15:30:49.084] ** Platform: Windows
[2026-08-13 15:30:49.085] ** Python version: 3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
[2026-08-13 15:30:49.085] ** Python executable: D:\ai\ComfyUI\venv\Scripts\python.exe
[2026-08-13 15:30:49.085] ** ComfyUI Path: D:\ai\ComfyUI
[2026-08-13 15:30:49.085] ** ComfyUI Base Folder Path: D:\ai\ComfyUI
[2026-08-13 15:30:49.086] ** User directory: D:\ai\ComfyUI\user
[2026-08-13 15:30:49.086] ** ComfyUI-Manager config path: D:\ai\ComfyUI\user\__manager\config.ini
[2026-08-13 15:30:49.086] ** Log path: D:\ai\ComfyUI\user\comfyui.log
Prestartup times for custom nodes:
[2026-08-13 15:30:54.869] 0.0 seconds: D:\ai\ComfyUI\custom_nodes\comfyui-easy-use
[2026-08-13 15:30:54.870] 0.0 seconds: D:\ai\ComfyUI\custom_nodes\rgthree-comfy
[2026-08-13 15:30:54.871] 13.9 seconds: D:\ai\ComfyUI\custom_nodes\comfyui-manager
[2026-08-13 15:30:54.872]
[2026-08-13 15:31:12.873] D:\ai\ComfyUI\venv\Lib\site-packages\torch\cuda\__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
[2026-08-13 15:31:19.269] Found comfy_kitchen backend cuda: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'dequantize_w4a8_int8_weight', 'gemv_awq_w4a16', 'int8_linear', 'na3d', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'quantize_w4a8_int8_weight', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'rotate_int8_convrot_weight', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8', 'w4a8_int8_linear']}
[2026-08-13 15:31:19.269] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_embedding', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'dequantize_w4a8_int8_weight', 'gemv_awq_w4a16', 'int8_linear', 'na3d', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'quantize_w4a8_int8_weight', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'rotate_int8_convrot_weight', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8', 'w4a8_int8_linear']}
[2026-08-13 15:31:19.269] Found comfy_kitchen backend hip: {'available': False, 'disabled': False, 'unavailable_reason': 'PyTorch ROCm/HIP runtime not available', 'capabilities': []}
[2026-08-13 15:31:19.270] Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'int8_linear', 'na3d', 'quantize_and_rotate_rowwise', 'quantize_int8_rowwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'w4a8_int8_linear']}
[2026-08-13 15:31:19.312] Checkpoint files will always be loaded safely.
[2026-08-13 15:31:19.435] Total VRAM 16376 MB, total RAM 97918 MB
[2026-08-13 15:31:19.436] pytorch version: 2.12.0+cu132
[2026-08-13 15:31:19.436] Set vram state to: NORMAL_VRAM
[2026-08-13 15:31:19.436] Device: cuda:0 NVIDIA GeForce RTX 4090 Laptop GPU : cudaMallocAsync
[2026-08-13 15:31:19.462] Using async weight offloading with 2 streams
[2026-08-13 15:31:19.463] Enabled pinned memory 39167.0
[2026-08-13 15:31:19.651] Using pytorch attention
[2026-08-13 15:32:02.494] aimdo: src-win/cuda-detour.c:38:INFO:aimdo_setup_hooks: installing 6 hooks
[2026-08-13 15:32:02.523] aimdo: src/control.c:262:INFO:comfy-aimdo NVML pressure enabled
[2026-08-13 15:32:02.534] aimdo: src-win/shmem-detect.c:80:INFO:comfy-aimdo WDDM adapter match: NVIDIA GeForce RTX 4090 Laptop GPU runtime_luid=00000000:00014288 dxgi_luid=00000000:00014288
[2026-08-13 15:32:02.534] aimdo: src/control.c:277:INFO:comfy-aimdo inited for GPU: NVIDIA GeForce RTX 4090 Laptop GPU (VRAM: 16375 MB)
[2026-08-13 15:32:02.534] DynamicVRAM support detected and enabled
[2026-08-13 15:32:02.534] Python version: 3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
[2026-08-13 15:32:02.534] ComfyUI version: 0.32.0
[2026-08-13 15:32:02.535] comfy-aimdo version: 0.4.13
[2026-08-13 15:32:02.536] comfy-kitchen version: 0.2.30
[2026-08-13 15:32:02.539] comfyui-frontend-package version: 1.48.7
[2026-08-13 15:32:02.539] comfyui-workflow-templates version: 0.11.40
[2026-08-13 15:32:02.539] comfyui-embedded-docs version: 0.5.9
[2026-08-13 15:32:02.539] comfy-kitchen version: 0.2.30
[2026-08-13 15:32:02.539] comfy-aimdo version: 0.4.13
[2026-08-13 15:32:02.540] [Prompt Server] web root: D:\ai\ComfyUI\venv\Lib\site-packages\comfyui_frontend_package\static
[2026-08-13 15:32:02.541] Asset seeder disabled
[2026-08-13 15:32:06.096] No OpenGL_accelerate module loaded: Acceleration disabled
[2026-08-13 15:32:08.150] Web extensions folder found at D:\ai\ComfyUI\web\extensions\ComfyLiterals
[2026-08-13 15:32:10.478] WARNING: BNB_CUDA_VERSION=130 environment variable detected; loading libbitsandbytes_cuda130.dll.
This can be used to load a bitsandbytes version built with a CUDA version that is different from the PyTorch CUDA version.
... [custom node loading omitted] ...
[2026-08-13 15:34:26.333] got prompt
[2026-08-13 15:34:28.951] Using pytorch attention in VAE
[2026-08-13 15:34:28.959] Using pytorch attention in VAE
[2026-08-13 15:34:29.405] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[2026-08-13 15:34:29.559] FETCH ComfyRegistry Data: 170/171
[2026-08-13 15:34:30.875] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[2026-08-13 15:34:30.882] Requested to load Krea2TEModel_
[2026-08-13 15:34:31.358] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[2026-08-13 15:34:31.502] FETCH ComfyRegistry Data [DONE]
[2026-08-13 15:34:31.707] [ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
[2026-08-13 15:34:31.755] FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json [DONE]
[2026-08-13 15:34:31.913] [ComfyUI-Manager] All startup tasks have been completed.
[2026-08-13 15:34:33.704] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[2026-08-13 15:34:33.991] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[2026-08-13 15:34:34.308] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
... [~90 repeats of 'Model Krea2TEModel_ prepared for dynamic VRAM loading' omitted] ...
[2026-08-13 15:35:00.691] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[2026-08-13 15:35:00.992] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[2026-08-13 15:35:01.307] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[2026-08-13 15:35:01.626] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[2026-08-13 15:35:01.949] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[2026-08-13 15:35:02.261] Model Krea2TEModel_ prepared for dynamic VRAM loading. 8463MB Staged. 0 patches attached. Force pre-loaded 249 weights: 627 KB.
[2026-08-13 15:35:02.523] Found quantization metadata version 1
[2026-08-13 15:35:02.523] Detected mixed precision quantization
[2026-08-13 15:35:02.524] Using mixed precision operations
[2026-08-13 15:35:02.524] Native ops: asym_w4a8_int8, convrot_w4a4, float8_e4m3fn, float8_e5m2, int8_tensorwise , emulated ops: nvfp4, mxfp8
[2026-08-13 15:35:02.540] model weight dtype torch.bfloat16, manual cast: torch.bfloat16
[2026-08-13 15:35:02.544] model_type FLUX
[2026-08-13 15:35:07.096] Requested to load Krea2
[2026-08-13 15:35:07.211] Model Krea2 prepared for dynamic VRAM loading. 12864MB Staged. 256 patches attached. Force pre-loaded 160 weights: 2824 KB.
[2026-08-13 15:35:20.457]
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:09<00:00, 1.30s/it]
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:09<00:00, 1.14s/it]
[2026-08-13 15:35:21.172] Requested to load Krea2
[2026-08-13 15:35:21.264] Model Krea2 prepared for dynamic VRAM loading. 12864MB Staged. 256 patches attached. Force pre-loaded 160 weights: 2824 KB.
[2026-08-13 15:35:33.230]
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:11<00:00, 1.32s/it]
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:11<00:00, 1.38s/it]
[2026-08-13 15:35:33.874] Requested to load Krea2
[2026-08-13 15:35:33.960] Model Krea2 prepared for dynamic VRAM loading. 12864MB Staged. 256 patches attached. Force pre-loaded 160 weights: 2824 KB.
[2026-08-13 15:35:46.227]
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:11<00:00, 1.38s/it]
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:11<00:00, 1.43s/it]
[2026-08-13 15:35:46.825] Requested to load Krea2
... [iterations 3..16 identical: 'Requested to load Krea2' / '12864MB Staged. 256 patches attached.' / 8 steps @ ~1.40s/it] ...
[2026-08-13 15:38:34.607] Requested to load Krea2
[2026-08-13 15:38:34.690] Model Krea2 prepared for dynamic VRAM loading. 12864MB Staged. 256 patches attached. Force pre-loaded 160 weights: 2824 KB.
[2026-08-13 15:38:47.278]
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:11<00:00, 1.46s/it]
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:11<00:00, 1.47s/it]
[2026-08-13 15:38:47.900] Requested to load Krea2
[2026-08-13 15:38:48.006] Model Krea2 prepared for dynamic VRAM loading. 12864MB Staged. 256 patches attached. Force pre-loaded 160 weights: 2824 KB.
[2026-08-13 15:38:49.269] aimdo: src/hostbuf.c:46:ERROR:hostbuf_grow: requested 26999484416 bytes beyond reserved host buffer 26998145024
[2026-08-13 15:38:49.274] aimdo: src/hostbuf.c:46:ERROR:hostbuf_grow: requested 26999484416 bytes beyond reserved host buffer 26998145024
[2026-08-13 15:38:49.372] aimdo: src/hostbuf.c:46:ERROR:hostbuf_grow: requested 27001171968 bytes beyond reserved host buffer 26998145024
[2026-08-13 15:38:49.379] aimdo: src/hostbuf.c:46:ERROR:hostbuf_grow: requested 27009118208 bytes beyond reserved host buffer 26998145024
[2026-08-13 15:38:49.385] aimdo: src/hostbuf.c:46:ERROR:hostbuf_grow: requested 27009118208 bytes beyond reserved host buffer 26998145024
[2026-08-13 15:38:49.390] aimdo: src/hostbuf.c:46:ERROR:hostbuf_grow: requested 27009118208 bytes beyond reserved host buffer 26998145024
[2026-08-13 15:38:49.408] aimdo: src/hostbuf.c:46:ERROR:hostbuf_grow: requested 27001171968 bytes beyond reserved host buffer 26998145024
... [~600 more identical hostbuf_grow ERROR lines, sizes 26998939648 .. 27010842624, all vs the same 26998145024 cap] ...
[2026-08-13 15:39:31.201] Interrupting prompt 5acd35eb-347b-4521-8c62-50509be90180
[2026-08-13 15:39:31.203]
[2026-08-13 15:39:34.969] Processing interrupted
[2026-08-13 15:39:34.969] Prompt executed in 308.19 seconds
```
### Other
**Versions**
| | |
|---|---|
| ComfyUI | 0.32.0 (`bd34f33`) |
| comfy-aimdo | 0.4.13 |
| comfy-kitchen | 0.2.30 |
| frontend | 1.48.7 |
| torch | 2.12.0+cu132 |
| Python | 3.12.10 |
| OS | Windows 11 |
| GPU | RTX 4090 Laptop, 16375 MB VRAM |
| RAM | 97918 MB |
**Where the cap comes from**
`comfy/model_management.py:1575`:
```python
def pinned_hostbuf_size(size):
if args.high_ram:
return max(0, int(size * 2))
return max(0, int(min(size, MAX_PINNED_MEMORY) * 2))
```
`comfy/model_patcher.py:1876` allocates four buffers, all with that same cap, once per model:
```python
hostbuf_size = comfy.model_management.pinned_hostbuf_size(self.model_size())
pin_state["weights"] = (comfy_aimdo.host_buffer.HostBuffer(0, 64 * 1024 * 1024, hostbuf_size), ...)
pin_state["patches"] = (comfy_aimdo.host_buffer.HostBuffer(0, 8 * 1024 * 1024, hostbuf_size), ...)
pin_state["weights-loaded"] = (comfy_aimdo.host_buffer.HostBuffer(0, 64 * 1024 * 1024, hostbuf_size), ...)
pin_state["patches-loaded"] = (comfy_aimdo.host_buffer.HostBuffer(0, 8 * 1024 * 1024, hostbuf_size), ...)
```
`2 x model_size` is a fixed headroom that does not scale with how many times the model is re-staged in one prompt, and the buffer appears to be bump-allocated without reclaiming freed regions between `Requested to load Krea2` cycles. With 256 patches per reload that headroom is consumed in ~18 iterations.
**Two separate things that would each help**
1. Reclaim / reuse host buffer space between reloads of the *same* model within one prompt - the buffer should reach steady state, not grow with iteration count.
2. Rate-limit or dedupe the `hostbuf_grow` ERROR (log once + a Python-level warning), instead of 618 identical lines. Right now a real regression in the same session would be invisible.
**Possibly related, but not the same call site:** #14250, #15255 (both `HostBuffer.read_file_slice`, hard OOM failures). This one is `hostbuf_grow`, soft-fails, and is triggered by repetition rather than by model size.
Full log and workflow attached.
[comfyui.log](https://github.com/user-attachments/files/31034603/comfyui.log)
[XY plot - Prompt-LoraStrength.json](https://github.com/user-attachments/files/31034687/XY.plot.-.Prompt-LoraStrength.json)
Contributor guide
Assessment
This issue has not been assessed yet.