prompt_worker thread dies on any exception escaping `PromptExecutor.execute` — server keeps serving HTTP but never executes prompts again ("zombie" server)
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
### Expected Behavior
## Summary
`prompt_worker()` in `main.py` calls `e.execute(...)` without any exception guard. If an exception escapes `PromptExecutor.execute_async` from *framework-level* code (anything outside the per-node `try/except` that turns node errors into `ExecutionResult.FAILURE`), the exception propagates out of `e.execute()` and kills the `prompt_worker` thread.
After that the server is in a zombie state:
- the HTTP server keeps responding normally (`/`, `/queue`, `/history`, `/prompt` all return 200),
- every newly queued prompt is accepted but never executed,
- there is no error in the UI — jobs just hang forever, and the only recovery is restarting ComfyUI.
This matches the traceback pattern of several "ComfyUI stops executing prompts" reports (e.g. AMD/HIP illegal-memory-access crash logs where the final line is `Exception in thread Thread-N (prompt_worker)` followed by a silent server).
## Real-world trigger (ComfyUI 0.35.1, Windows, 100% reproducible)
We hit a deterministic instance of this with the RAM-pressure cache:
1. A workflow contains a node whose output includes a **lazily-materialized Mapping** (VideoHelperSuite's `LazyAudioMap`, which shells out to ffmpeg on first access). The lazy object was never consumed during execution, but it gets stored in the output cache.
2. After the prompt finished, our automation deleted the input file that the lazy object references.
3. When the **next** prompt starts, `execute_async` calls `ram_release_callback(...)` (RAM-pressure cache eviction). The eviction scan iterates cached outputs, materializes the lazy Mapping → ffmpeg fails (`No such file or directory`) → exception.
4. `execute_async`'s `try` block has **no `except` clause** (only `finally`), so the exception escapes `e.execute()` and kills `prompt_worker`:
```
Exception in thread Thread-9 (prompt_worker):
Traceback (most recent call last):
File "main.py", line 394, in prompt_worker
e.execute(item[2], prompt_id, extra_data, item[4])
File "execution.py", line 728, in execute
asyncio.run(self.execute_async(prompt, prompt_id, extra_data, execute_outputs))
File "execution.py", line 800, in execute_async
ram_release_callback(ram_inactive_headroom)
File "comfy_execution/caching.py", line 591, in ram_release
scan_list_for_ram_usage(cache_entry.outputs)
File "custom_nodes/ComfyUI-VideoHelperSuite/videohelpersuite/utils.py", line 262, in __iter__
self._dict = get_audio(self.file, self.start_time, self.duration)
...
Exception: VHS failed to extract audio from .../input/lip_shot_004.mp4: ...
Error opening input files: No such file or directory
```
From this point on every prompt submitted to the server is silently never executed. Deleting the database / cache does not help; only a restart does.
The same class of failure can be triggered by any other framework-level exception escaping `execute_async` (cache providers, asset enrichment, progress handlers, ...), so the fix is not specific to VHS.
## Proposed fixes (two small patches, both against current master)
### 1. `main.py` — the worker thread must never die
Wrap `e.execute(...)` in `try/except`, log the exception, and record the prompt as failed so `q.task_done(...)` still completes the queue item with an error status. `history_result` may not exist if `execute_async` failed early, hence the `getattr` fallback.
```patch
--- a/main.py
+++ b/main.py
@@ -359,7 +359,19 @@
extra_data[k] = sensitive[k]
asset_manager.pause_background_scan()
- e.execute(item[2], prompt_id, extra_data, item[4])
+ try:
+ e.execute(item[2], prompt_id, extra_data, item[4])
+ except Exception:
+ # Any exception escaping the executor (e.g. from cache
+ # maintenance or other framework-level code outside the
+ # per-node error handling) must not kill the prompt_worker
+ # thread: the HTTP server would keep running but no prompt
+ # would ever execute again ("zombie" server).
+ logging.exception(
+ "Unhandled exception escaped PromptExecutor.execute; "
+ "recording the prompt as failed and keeping prompt_worker alive")
+ e.success = False
+ e.history_result = getattr(e, "history_result", None) or {}
need_gc = True
```
### 2. `comfy_execution/caching.py` — `ram_release` must tolerate scan failures
A cache entry whose outputs fail to scan (e.g. lazy materialization error) should not crash the executor. Treat it as having unknown-but-nonzero size so it still participates in eviction.
```patch
--- a/comfy_execution/caching.py
+++ b/comfy_execution/caching.py
@@ -1,6 +1,7 @@
import asyncio
import bisect
import itertools
+import logging
import time
import torch
from typing import Sequence, Mapping, Dict
@@ -588,7 +589,23 @@
oom_ram_usage = 1e30
elif hasattr(output, "_comfy_cache_tensors"):
scan_list_for_ram_usage(output._comfy_cache_tensors())
- scan_list_for_ram_usage(cache_entry.outputs)
+ try:
+ scan_list_for_ram_usage(cache_entry.outputs)
+ except Exception:
+ # Cached outputs may contain lazily-materialized objects (e.g.
+ # a custom node's lazy Mapping that shells out to ffmpeg on
+ # first access). If materialization fails (e.g. the referenced
+ # file was deleted after the workflow ran), the exception must
+ # not escape ram_release(): it is called from execute_async
+ # outside the per-node error handling, so it would kill the
+ # prompt_worker thread and leave the server "zombie" (HTTP
+ # alive, queue dead). Treat the entry as having unknown size so
+ # it still participates in eviction.
+ logging.exception(
+ "ram_release: failed to scan cached outputs; "
+ "treating cache entry as evictable")
+ ram_usage = max(ram_usage, min_entry_size)
+ oom_ram_usage = max(oom_ram_usage, ram_usage)
if ram_usage < min_entry_size:
continue
```
## Verification
Both patches applied locally on 0.35.1: the exact trigger above (lipsync automation deleting its uploaded input file between prompts) previously turned the server into a zombie **every time**; with the patches the scan failure is logged, the entry is treated as evictable, and the server keeps executing prompts.
## Environment
- ComfyUI 0.35.1 (vulnerable code paths unchanged on master as of today: `main.py` `prompt_worker` bare `e.execute(...)`, `caching.py` `ram_release` unguarded `scan_list_for_ram_usage(cache_entry.outputs)`, `execution.py` `execute_async` `try` with only `finally`)
- Windows 11, Python 3.12, RTX 5090D
### Actual Behavior
The prompt_worker thread dies, leaving the server in a "zombie" state:
- The HTTP server keeps responding normally — `/`, `/queue`, `/history`, `/prompt` all return 200.
- Every newly queued prompt is accepted but **never executed**. Jobs hang forever in the queue.
- No error is shown in the UI. The only visible sign is the terminal traceback (`Exception in thread Thread-N (prompt_worker)`, see Debug Logs).
- Deleting the database / cache does not help. Only restarting ComfyUI recovers.
In our production automation this made ComfyUI look "alive" (health checks passed) while nothing ever ran again — the failure silently blocked all rendering until we correlated restarts with the terminal traceback.
### Steps to Reproduce
1. Start ComfyUI 0.35.1 (Windows) with the RAM-pressure cache enabled (`--cache ram`).
2. Run a workflow whose output contains a **lazily-materialized Mapping** — e.g. any VideoHelperSuite audio node (`LazyAudioMap` shells out to ffmpeg on first access). The lazy object is stored in the output cache even if never consumed.
3. After the prompt finishes, delete the input file that the lazy object references (in our automation: the uploaded `input/lip_shot_004.mp4` is removed between prompts).
4. Queue any new prompt. On startup of that prompt, `execute_async` calls `ram_release_callback(...)`; the eviction scan iterates cached outputs, materializes the lazy Mapping, ffmpeg fails (`No such file or directory`), and the exception escapes `e.execute()` because `prompt_worker()` has no guard — killing the thread.
**Expected:** the failed cache scan is logged, the prompt runs (or is recorded as failed), and subsequent prompts keep executing.
**Actual:** the HTTP server keeps responding normally (`/`, `/queue`, `/history`, `/prompt` all return 200), but every newly queued prompt is accepted and never executed. No error appears in the UI; jobs hang forever. Only a restart of ComfyUI recovers. Deleting the database / cache does not help.
Note: no minimal workflow JSON is attached because the trigger is the *automation step* (deleting a referenced input file between prompts), not the workflow graph itself — any workflow containing a VHS audio node reproduces it. The failure class is also not VHS-specific: any framework-level exception escaping `execute_async` (cache providers, asset enrichment, progress handlers, ...) kills `prompt_worker` the same way.
### Debug Logs
```powershell
Exception in thread Thread-9 (prompt_worker):
Traceback (most recent call last):
File "main.py", line 394, in prompt_worker
e.execute(item[2], prompt_id, extra_data, item[4])
File "execution.py", line 728, in execute
asyncio.run(self.execute_async(prompt, prompt_id, extra_data, execute_outputs))
File "execution.py", line 800, in execute_async
ram_release_callback(ram_inactive_headroom)
File "comfy_execution/caching.py", line 591, in ram_release
scan_list_for_ram_usage(cache_entry.outputs)
File "custom_nodes/ComfyUI-VideoHelperSuite/videohelpersuite/utils.py", line 262, in __iter__
self._dict = get_audio(self.file, self.start_time, self.duration)
...
Exception: VHS failed to extract audio from .../input/lip_shot_004.mp4: ...
Error opening input files: No such file or directory
(After this traceback the HTTP server keeps responding normally, but every
newly queued prompt is accepted and never executed. Only a restart recovers.)
```
### Other
_No response_
Contributor guide
Research direction
Start in main.py at prompt_worker and then read execution.py around execute_async and comfy_execution/caching.py around ram_release. Reproduce with --cache ram and a lazily materialized output whose input is deleted between prompts. Done means scan failures are logged or recorded as failed without killing the worker, and subsequent prompts continue executing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100