Comfy-Org / Comfy-Org/comfy-aimdo

Segfault in `hostbuf_free()` during RAM-pressure cache eviction (`HostBuffer.__del__`)

Open
#97 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
67
Forks
39
Avg merge
1d 25m
Merged PRs (30d)
10

Description

## Summary

A ComfyUI instance running with DynamicVRAM enabled dies with `SIGSEGV` inside
`aimdo.so`'s `hostbuf_free()`. The crash is reached from `HostBuffer.__del__`,
which is invoked when ComfyUI's RAM-pressure cache evicts a stale cache entry
holding the last reference to a `ModelPatcherDynamic` (and therefore to its
`dynamic_pins` `HostBuffer` objects).

The process is killed outright — `faulthandler` prints the Python-level stack,
then the interpreter is gone. It has happened twice on the same host, both times
about 2 seconds after a new prompt was submitted, i.e. at the first
post-node-execution eviction of the previous prompt's cached models.

`comfy-aimdo` 0.4.14 does **not** fix it: its `aimdo.so` is byte-identical to
0.4.13 (see *Ruled out* below).

## Environment

| Item | Value |
| ------------- | ------------------------------------------------------------ |
| comfy-aimdo | 0.4.13 (`aimdo.so` md5 `8a58ab558fac0b291cd4dbe1730947b7`, Build ID `1771a215de85c1a92a27615f7336a42cf7932e6f`) |
| ComfyUI | 0.33.0 |
| Python | 3.11.13 (conda) |
| torch | 2.8.0+cu128 |
| xformers | 0.0.32.post2 |
| numpy | 1.26.3 |
| psutil | 6.1.1 |
| OS | Ubuntu 24.04.1 LTS, kernel 6.11.0-29-generic |
| NVIDIA driver | 560.28.03 |
| GPU | 2 × RTX 4090 24 GB (sm_89); one ComfyUI instance pinned per GPU via `CUDA_VISIBLE_DEVICES` |
| Host RAM | 61 GiB total, 109 GiB swap |

Launch arguments of the crashing instance (GPU 0):

```
python main.py --listen --port 8190 --disable-pinned-memory \
--models-directory --extra-model-paths-config \
--database-url sqlite:////comfyui_gpu0.db --temp-directory /gpu0
```

Note there is **no** `--cache-*` flag, so the default RAM-pressure cache
(`CacheType.RAM_PRESSURE`) is in effect.

## Crash output

Identical stack on both occurrences (2026-08-24 15:03:27 CST, PID 2115801;
2026-08-25 15:17:01 CST, PID 2415539):

```
[INFO] got prompt
[INFO] current client id:
Fatal Python error: Segmentation fault

Stack (most recent call first):
File ".../site-packages/comfy_aimdo/host_buffer.py", line 129 in __del__
File ".../ComfyUI/comfy_execution/caching.py", line 593 in ram_release
File ".../ComfyUI/execution.py", line 800 in execute_async
File ".../python3.11/asyncio/events.py", line 84 in _run
File ".../python3.11/asyncio/base_events.py", line 1936 in _run_once
File ".../python3.11/asyncio/base_events.py", line 608 in run_forever
File ".../python3.11/asyncio/base_events.py", line 641 in run_until_complete
File ".../python3.11/asyncio/runners.py", line 118 in run
File ".../python3.11/asyncio/runners.py", line 190 in run
File ".../ComfyUI/execution.py", line 728 in execute
File ".../ComfyUI/main.py", line 450 in prompt_worker
File ".../python3.11/threading.py", line 982 in run
File ".../python3.11/threading.py", line 1045 in _bootstrap_inner
File ".../python3.11/threading.py", line 1002 in _bootstrap
Extension modules: <321 modules>
```

`host_buffer.py:129` is the `lib.hostbuf_free(ptr)` call in `HostBuffer.__del__`:

```python
def __del__(self):
ptr = getattr(self, "_ptr", None)
if ptr:
if lib is not None:
lib.hostbuf_free(ptr) # <-- line 129, SIGSEGV here
self._ptr = None
```

## How the crash path is reached

Reading the ComfyUI 0.33.0 source along the stack:

1. `execution.py:799-811` — after **every** node execution, when
`cache_type == CacheType.RAM_PRESSURE`, the executor calls
`ram_release_callback(ram_inactive_headroom)`.

2. `comfy_execution/caching.py:550` `RAMPressureCache.ram_release()` builds a
candidate list of cache entries whose `used_generation != generation`
(i.e. not touched by the current prompt) and assigns each an OOM score.
Entries holding a `ModelPatcher` from an older generation are given
`oom_ram_usage = 1e30` with the comment *"old ModelPatchers are the first to
go"*, so they sort to the top of the eviction list.

3. `caching.py:591-597` — the eviction loop `del self.cache[key]` drops the last
reference to that `ModelPatcherDynamic`. Its
`model.dynamic_pins[device]` pin-state tuples (created at
`comfy/model_patcher.py:1877-1880` and `comfy/model_management.py:1453`) each
hold a `comfy_aimdo.host_buffer.HostBuffer`, so Python refcounting runs
`HostBuffer.__del__` → `lib.hostbuf_free()` → SIGSEGV.

### Why this path runs constantly on this host

`main.py:412-417` computes the two RAM-pressure thresholds from total RAM:

```python
cache_ram = min(10.0, max(2.0, total_ram_mb * 0.10 / 1024.0)) # ~6.2 GiB
cache_ram_inactive = min(128.0, total_ram_mb / 1024.0) # ~62.0 GiB
```

`ram_release()` returns early only when `psutil.virtual_memory().available >= target`.
With `ram_inactive_headroom ≈ 62 GiB` on a 61 GiB machine, that condition can
never be satisfied, so the eviction loop executes after *every single node* and
frees every inactive entry it can find. On this host `available` sits around
32 GiB, so the loop always runs to completion.

That matches the observed timing exactly: both crashes landed ~2 s after
`got prompt`, before any model-loading log line — i.e. at the first eviction
following the first node of a new prompt, when the *previous* prompt's
`ModelPatcherDynamic` objects became stale.

## Ruled out

**Not an OOM kill.** No `oom-kill` / `Killed process` entries in the kernel log
or journal; at crash time the host had ~32 GiB RAM available and 103 GiB of
109 GiB swap free. The process died from `SIGSEGV`, and `faulthandler` printed a
Python stack, which an OOM kill would not produce.

**Not fixed in 0.4.14.** The only release newer than 0.4.13 ships the same
native library:

```
$ pip download --no-deps comfy-aimdo==0.4.14
$ md5sum {installed-0.4.13,downloaded-0.4.14}/comfy_aimdo/aimdo.so
8a58ab558fac0b291cd4dbe1730947b7 0.4.13/comfy_aimdo/aimdo.so
8a58ab558fac0b291cd4dbe1730947b7 0.4.14/comfy_aimdo/aimdo.so
$ diff -rq 0.4.13/comfy_aimdo 0.4.14/comfy_aimdo
Files ... _version.py differ # only file that differs
```

`host_buffer.py` is identical between the two versions.

**Not simply "freeing a degenerate buffer".** Because the instance runs with
`--disable-pinned-memory`, `MAX_PINNED_MEMORY` stays at its initial `-1`
(`comfy/model_management.py:1562`), so

```python
def pinned_hostbuf_size(size): # model_management.py:1592
return max(0, int(min(size, MAX_PINNED_MEMORY) * 2)) # -> 0
```

returns `0`, and the pin-state buffers are constructed as
`HostBuffer(0, 64 MiB, 0)` / `HostBuffer(0, 8 MiB, 0)` — a non-zero `prewarm`
with `max_mmap_size == 0`. We tested that combination in isolation:

```python
control.init()
b = HostBuffer(0, 64 * 1024**2, 0) # allocates, returns non-null ptr
del b # -> "FREED_OK", no crash
```

All of `(0,0,0)`, `(0,8 MiB,0)`, `(0,64 MiB,0)`, `(0,8 MiB,2 GiB)` and
`(0,64 MiB,2 GiB)` allocate and free cleanly in a fresh process. So the crash
needs additional state — a buffer that has actually been `extend()`ed and
`register()`ed, with live CUDA host registrations or in-flight transfers at the
moment of free. That state is not reachable from a standalone script, which is
why we cannot offer a self-contained reproducer.

## Frequency and impact

Two crashes in two days on the GPU 0 instance. A second instance on the same
host with an identical command line (only port, GPU, DB path, temp dir and
`--disable-metadata` differ) has run since 2026-08-19 without crashing, so the
trigger appears workload-dependent rather than configuration-dependent — both
instances execute the same `ram_release` path on every node.

The workload preceding the 2026-08-25 crash was a Flux2 graph using dynamic VRAM
staging, e.g.:

```
Model Flux2TEModel_ prepared for dynamic VRAM loading. 8262MB Staged.
Model Flux2 prepared for dynamic VRAM loading. 17316MB Staged.
Model AutoencoderKL prepared for dynamic VRAM loading. 160MB Staged.
Prompt executed in 74.97 seconds
```

Impact is larger than a single failed prompt: the service is under
`Restart=always`, and reloading ~4260 custom nodes takes ~6 min 48 s before the
HTTP port is bound again, so each crash costs about 7 minutes of downtime.

## Suspected cause

We cannot see inside `aimdo.so`, but the evidence points at
`hostbuf_free()` releasing host memory that is still referenced by the CUDA
driver — either a still-registered range (no `cudaHostUnregister` before the
munmap/free) or a buffer whose decommit is racing an async copy. ComfyUI's own
comment at `execution.py:807` acknowledges asynchronous decommit behaviour:

```python
# AIMDO MEM_DECOMMIT can outrun psutil.available catching up.
time.sleep(0.05)
```

A defensive `hostbuf_free()` that unregisters and synchronises before releasing —
or that tolerates being called on a buffer whose registrations were already torn
down — would likely stop the process from dying.

## What we can provide

The host is still running and the fault is reproducible over a timescale of
days. On request we can supply:

- `comfy_aimdo.control.set_log_debug()` output leading up to a crash
(ComfyUI exposes this via `--verbose DEBUG`).
- A core dump with `aimdo.so` symbols, if a debug build or symbol file is
published — the shipped `.so` is stripped apart from its Build ID.
- The exact workflow JSON, under NDA-free redaction if needed.

## Secondary observation (ComfyUI side, not aimdo)

`pinned_hostbuf_size()` silently collapses to `0` when `--disable-pinned-memory`
is set, because `MAX_PINNED_MEMORY` remains `-1` and `min(size, -1) * 2` is
negative before the `max(0, ...)` clamp. The pin-state buffers are then created
with a non-zero `prewarm` but a zero `max_mmap_size`. That combination does not
crash on its own (tested above), but it is almost certainly not the intended
configuration and may be worth guarding on the ComfyUI side.

Contributor guide

Open the contributing guide

Research direction

Start with comfy_aimdo/host_buffer.py::__del__ and trace the eviction path through comfy_execution/caching.py:550-597 and execution.py:799-811. Enable comfy_aimdo debug logging and, if available, inspect a core dump while reproducing the RAM-pressure eviction workload. Done means the native host-buffer lifecycle is understood and the reported SIGSEGV no longer occurs during eviction.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
ai-infra-agents, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.