# Dynamic VRAM: reused (warm) model produces NaN/black output on VAE decode, fresh load does not

**After more research issue seem related to dynamic VRAM
https://github.com/Comfy-Org/ComfyUI/issues/15452#issuecomment-5234467521**

### Custom Node Testing

- [ ] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

## Expected behavior

All 4 images render correctly, identical in reliability to a fresh model load.

Hello,
### Actual Behavior

## Summary

With `--enable-dynamic-vram` active, running multiple prompts back-to-back against the **Boogu T2I Turbo** official template (a mixed-precision-quantized checkpoint) produces a fully black image on any generation where the diffusion model is *reused* from an already-resident state, but produces a correct image whenever the model is *freshly* loaded. VAE precision (`--fp16-vae` / `--bf16-vae` / `--fp32-vae`) does not affect the outcome — the same failure occurs regardless of VAE dtype.

The failure is deterministic and 100% reproducible in a 4-image batch (different seeds, same workflow): images 1 and 3 (fresh model load) came out clean; images 2 and 4 (warm/reused model) came out black, with a `RuntimeWarning: invalid value encountered in cast` at the final `Image.fromarray` step, indicating NaN/Inf in the decoded tensor.

## Environment

- ComfyUI version: `v0.31.0-8-gcbbc9dab` (released 2026-08-08)
- comfy-aimdo version: `0.4.13`
- comfy-kitchen version: `0.2.28`
- comfyui-frontend-package: `1.48.7`
- Python: `3.13.15`
- PyTorch: `2.12.0+rocm7.14.0`
- ROCm version: `7.14`
- GPU: AMD Radeon RX 7600 XT (gfx1102), 16368 MB VRAM
- CPU: AMD Ryzen 7 5700X3D (8-core)
- System RAM: 30986 MB
- OS: Linux 7.0.0-29-generic
- Launch flags: `--listen 0.0.0.0 --enable-dynamic-vram --fp32-vae`
- Model: Boogu T2I Turbo, official ComfyUI template workflow (unmodified), 4-step turbo sampling
- Model uses mixed-precision quantization: `Native ops: asym_w4a8_int8, convrot_w4a4, int8_tensorwise`

## Steps to reproduce

1. Launch ComfyUI with `--enable-dynamic-vram` (with or without `--fp32-vae` — both affected).
2. Load the official Boogu T2I Turbo template workflow.
3. Queue 2 or more generations back-to-back (e.g. a batch of 4 with different seeds) without freeing models between them.
4. Observe: the first generation (fresh model load) renders correctly. Every subsequent generation that reuses the already-resident Boogu model (i.e. no `Requested to load Boogu` line precedes it in the log — it goes straight to `Model Boogu prepared for dynamic VRAM loading`) renders fully black.

## Expected behavior

All 4 images render correctly, identical in reliability to a fresh model load.

## Actual behavior

Alternating clean/black output, tracking exactly with fresh-load vs. warm-reuse of the model:

| # | Model load type (log evidence) | Result |
|---|---|---|
| 1 | `Requested to load Boogu` → `Model Boogu prepared for dynamic VRAM loading` | Clean |
| 2 | `Model Boogu prepared for dynamic VRAM loading` (no preceding `Requested to load Boogu`) | Black — NaN |
| 3 | `Requested to load Boogu` → `Model Boogu prepared for dynamic VRAM loading` | Clean |
| 4 | `Model Boogu prepared for dynamic VRAM loading` (no preceding `Requested to load Boogu`) | Black — NaN |

## Relevant log excerpt

```
[20:39:01.695] Requested to load Boogu
[20:39:01.806] Model Boogu prepared for dynamic VRAM loading. 10842MB Staged. 0 patches attached. Force pre-loaded 357 weights: 1664 KB.
[20:39:21.372] 100%|██████████| 4/4 [00:18<00:00, 4.55s/it]
[20:39:21.378] Requested to load AutoencodingEngine
[20:39:21.413] Model AutoencodingEngine prepared for dynamic VRAM loading. 319MB Staged.
[20:39:43.631] Prompt executed in 49.10 seconds <-- CLEAN, fresh model load

[20:39:43.977] [MultiGPU_Memory_Management] Triggering PromptExecutor cache reset. Reason: cpu_threshold_exceeded
[20:39:44.091] Model Boogu prepared for dynamic VRAM loading. 10842MB Staged. <-- no "Requested to load Boogu" this time
[20:40:01.988] 100%|██████████| 4/4 [00:17<00:00, 4.45s/it]
[20:40:02.021] Model AutoencodingEngine prepared for dynamic VRAM loading. 319MB Staged.
[20:40:23.610] .../nodes.py:1691: RuntimeWarning: invalid value encountered in cast
img = Image.fromarray(np.clip(i, 0, 255).astype(np.uint8))
[20:40:23.638] Prompt executed in 39.66 seconds <-- BLACK, warm/reused model
```

The same pattern repeats identically for images 3 and 4 later in the same session (see attached full log).

## Additional notes / ruled-out causes

- **Not a VAE precision issue.** Tested with `--fp16-vae`(default/implicit bf16), `--bf16-vae`, and `--fp32-vae`. All three show the same fresh-load-clean / warm-reuse-black pattern. `--fp32-vae` only appeared to "fix" the issue in earlier single-generation testing because that test happened to be a fresh model load.
- **Not caused by low VRAM/OOM on the failing runs.** No `Ran out of memory` / tiled-decode-fallback warning appears immediately before the NaN in this batch — the failure occurs on a plain (non-tiled) decode.
- **Disabling dynamic VRAM (`--disable-dynamic-vram`) is not a viable workaround on this hardware** — it removes dynamic VRAM's memory-pressure management entirely, and system RAM usage balloons to ~27.7 GB (out of 30 GB), triggering the Linux OOM killer to kill the ComfyUI process outright (`journalctl -k`: `Out of memory: Killed process ... anon-rss:27738436kB`). So this bug can't currently be avoided by turning dynamic VRAM off.
- A `[MultiGPU_Memory_Management] Triggering PromptExecutor cache reset` event (from the MultiGPU custom node, triggered by CPU RAM >85%) appeared once during testing, immediately before one of the warm-reuse+NaN runs — possibly relevant, possibly coincidental given RAM pressure was already elevated from having the 10GB text encoder + 10GB diffusion model resident. Flagging in case it's a useful lead, but the fresh-load-vs-reuse correlation held even for the warm-reuse case that did *not* follow a cache-reset event.

## Current workaround

Force a full model unload/reload between generations (e.g. Manager's "Unload Models") instead of queuing multiple prompts back-to-back. This is reliable but defeats the purpose of dynamic VRAM's fast warm-model reuse path.

## Suspected component

Given the log evidence (staging without a fresh "Requested to load", combined with the model's mixed-precision quantization: `asym_w4a8_int8`, `convrot_w4a4`, `int8_tensorwise`), the bug likely lives in the interaction between the dynamic VRAM warm-reuse path and comfy-aimdo/comfy-kitchen's quantized inference ops — e.g. a dequantization scale, zero-point, or scratch buffer that's computed once on first load and not correctly re-derived/reset on subsequent reuse.

### Steps to Reproduce

[boogu_batch_repro.log](https://github.com/user-attachments/files/30878600/boogu_batch_repro.log)

### Debug Logs

[comfyui_stdout.log](https://github.com/user-attachments/files/30878709/comfyui_stdout.log)

### Other

_No response_

Reproduce the Boogu T2I Turbo workflow with --enable-dynamic-vram and compare fresh-load and warm-reuse runs, including the provided logs. Start at nodes.py:1691 and trace the dynamic VRAM preparation path alongside the comfy-aimdo/comfy-kitchen quantized inference behavior. Done means repeated generations produce valid, non-black images without requiring a full model unload/reload.

Written by the indexing model from the issue text.