[Dynamic VRAM][ROCm/Windows] MiniMax H3 VideoVAE slowdown still reproducible on current master; forced unload stabilizes ~16s

## Update 2026-08-12 — reproduced again on current master

The earlier apparent resolution after #15446 was temporary. I can reproduce the intermittent MiniMax H3 VideoVAE slowdown again on current master (`62b3c94b`).

New A/B testing strengthens the original diagnosis: with normal Dynamic VRAM co-residency, recent full-workflow VideoVAE stages ranged from about **17.9 s to 38.8 s**. When I explicitly clear H3/AudioVAE/VideoVAE residency immediately before VideoVAE decode, **six consecutive full H3 runs are tightly clustered at 16.06–16.28 s** (mean ~16.22 s, population stddev ~0.07 s). A separate VAE-only forced-unload series is similarly flat at about 15.80–15.88 s.

This is distinct from #15453/#15456 because **no OOM is required**: the decode completes successfully, but can take over 2× longer while the H3 model remains co-resident. See the latest comment for the current environment and detailed measurements.

---

### Custom Node Testing

- [ ] I have not repeated this with `--disable-all-custom-nodes`, because the reproduction uses a tiny diagnostic passthrough node to force only the downstream VAE decode to re-execute from a cached H3 latent.
- The sampler does **not** run during the measured VAE-only repetitions. The diagnostic node either returns the latent unchanged, or calls only ComfyUI core `unload_all_models()` / `soft_empty_cache()` before returning it.

### Summary

On Windows 11 with an AMD Radeon RX 9070 XT 16 GB, MiniMax H3 VideoVAE decode becomes highly variable and much slower when Dynamic VRAM keeps the H3 diffusion model resident alongside the VideoVAE.

I isolated the VAE from sampling by caching one H3 latent and repeatedly re-running only the downstream VideoVAE decode.

With the existing Dynamic VRAM residency left intact, 5 consecutive VAE-only runs took:

```text
43.48 s
26.57 s
47.00 s
35.22 s
52.05 s
```

When I force all models to unload and call `soft_empty_cache()` immediately before the **same cached latent / same VideoVAE decode**, 5 consecutive runs become:

```text
16.02 s
15.91 s
15.89 s
16.01 s
16.19 s
```

So the forced-unload control is both much faster and almost perfectly stable (15.89–16.19 s), while the normal Dynamic VRAM co-resident state varies from 26.57–52.05 s.

This strongly suggests a Dynamic VRAM / VBAR residency or eviction problem during VAE decode rather than a VideoVAE compute/kernel performance problem.

### Environment

- OS: Windows 11
- GPU: AMD Radeon RX 9070 XT, 16 GB VRAM (`gfx1201`)
- System RAM: 64 GB
- ComfyUI: `0.31.0`, revision 186, commit/revision shown as `[00d02f28]`, released 2026-08-08
- Python: `3.12.10`
- PyTorch: `2.14.0a0+rocm10.1.0a20260806`
- ROCm reported by ComfyUI: `(7, 15)`
- comfy-aimdo: `0.4.13`
- comfy-kitchen: `0.2.28`
- Dynamic VRAM: enabled
- `--disable-async-offload` is enabled
- Debug logging: `--verbose DEBUG`
- Pinned memory is enabled in the logs quoted below. I also tested `--disable-pinned-memory` separately: it prevents excessive shared/system-memory usage on this machine, but **does not fix this VAE timing instability**.

Relevant startup log:

```text
Total VRAM 16304 MB, total RAM 65040 MB
pytorch version: 2.14.0a0+rocm10.1.0a20260806
AMD arch: gfx1201
ROCm version: (7, 15)
Device: cuda:0 AMD Radeon RX 9070 XT : native
Enabled pinned memory 26016.0
DynamicVRAM support detected and enabled
ComfyUI version: 0.31.0
comfy-aimdo version: 0.4.13
comfy-kitchen version: 0.2.28
```

MiniMax H3 model sizes reported by ComfyUI:

```text
MiniMaxH3 ~19995 MB staged
MiniMaxH3VideoVAE ~4965 MB staged
MiniMaxH3AudioVAE ~576 MB staged
```

### Reproduction: VAE-only, without forced unload

1. Generate one MiniMax H3 latent normally.
2. Keep the sampler output cached.
3. Insert a passthrough node between the sampler latent and the H3 VAE decode. Change only an integer `trigger` each queue so ComfyUI re-executes the downstream VAE while the sampler remains cached.
4. Queue 5 repetitions without restarting ComfyUI or re-running the sampler.

Diagnostic passthrough node:

```python
class ForceLatentRerun:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"latent": ("LATENT",),
"trigger": ("INT", {
"default": 0,
"min": 0,
"max": 2147483647,
"step": 1,
}),
}
}

RETURN_TYPES = ("LATENT",)
FUNCTION = "run"
CATEGORY = "debug"

def run(self, latent, trigger):
return (latent,)
```

Observed VAE-only prompt times:

```text
43.48 s
26.57 s
47.00 s
35.22 s
52.05 s
```

The sampler did not execute in these 5 runs.

During these runs the residency state is repeatedly approximately:

```text
MiniMaxH3 resident: 9376 MB
MiniMaxH3AudioVAE resident: 416 MB
MiniMaxH3VideoVAE resident: 4736 MB
Total VRAM for VBARs: 14528 MB
Aimdo Recorded Usage: 14580 MB
Device free before/after: ~818 MB
```

During actual decode the device free memory also reaches zero:

```text
Aimdo Recorded Usage: 14580 MB
Device: 0 MB / 16304 MB Free

VBAR ... MiniMaxH3: Actual Resident VRAM = 9376 MB
VBAR ... AudioVAE: Actual Resident VRAM = 416 MB
VBAR ... MiniMaxH3VideoVAE: Actual Resident VRAM = 4736 MB
Total VRAM for VBARs: 14528 MB
```

Debug logging also repeatedly reports allocation pressure such as:

```text
budget_deficit: Prevailing Method: unknown Deficit: 27 Extra Headroom: 0 Alloc Size 32
VBAR allocator attempt exceeds available VRAM ...
```

Importantly, the VideoVAE is already fully resident (`vram_mb=4736.3`) in these VAE-only repeats, yet decode still varies by almost 2x.

### Control: force unload immediately before the same VAE decode

I then replaced the passthrough with this diagnostic node:

```python
import comfy.model_management as mm

class ForceUnloadBeforeVAE:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"latent": ("LATENT",),
"trigger": ("INT", {
"default": 0,
"min": 0,
"max": 2147483647,
"step": 1,
}),
}
}

RETURN_TYPES = ("LATENT",)
FUNCTION = "run"
CATEGORY = "debug"

def run(self, latent, trigger):
mm.unload_all_models()
mm.soft_empty_cache()
return (latent,)
```

Again, the sampler output remains cached. Only VAE decode is measured.

Observed 5 consecutive VAE-only times:

```text
16.02 s
15.91 s
15.89 s
16.01 s
16.19 s
```

Before each VAE run, the logs confirm that the previous VBAR residency is cleared:

```text
AIMDO free: model=MiniMaxH3VideoVAE ... requested=all vbar_mb=4736.0
Unloading MiniMaxH3VideoVAE
Aimdo Recorded Usage: 52 MB
Device: 15346 MB / 16304 MB Free
Total VRAM for VBARs: 0 MB
```

After loading the VideoVAE by itself, there is plenty of headroom:

```text
MiniMaxH3VideoVAE: Actual Resident VRAM = 4736 MB
Total VRAM for VBARs: 4736 MB
Aimdo Recorded Usage: 4788 MB
Device: 10610 MB / 16304 MB Free
Prompt executed in 15.89 seconds
```

### Full-workflow observation

The issue originally appeared as an intermittent slow VideoVAE stage after H3 sampling. One slow case reached VideoVAE with WDDM almost exactly at budget:

```text
WDDM budget=15416 MB usage=15320 MB
Device free=2955 MB

MiniMaxH3 resident: 12384 MB
MiniMaxH3AudioVAE resident: 416 MB
Total VBAR: 12800 MB
Requested to load MiniMaxH3VideoVAE
```

Full-workflow VAE timing was inconsistent, which is why I isolated the VAE-only reproduction above.

### Expected Behavior

Dynamic VRAM should automatically make enough usable VRAM/headroom available for VideoVAE activations/workspaces when transitioning from H3 sampling to VAE decode, without severe intermittent paging stalls.

I would expect repeated decode of the exact same cached latent to stay reasonably close to the ~16 s forced-unload baseline, rather than randomly varying between ~27 and ~52 s.

### Actual Behavior

Dynamic VRAM keeps a large amount of the H3 model resident while the VideoVAE is also resident. The combined VBAR residency is ~14.5 GB on a 16.3 GB device, and actual device free memory reaches 0 MB during decode.

At that point Aimdo reports repeated budget deficits / VBAR allocation attempts, and the exact same VAE decode varies from 26.57–52.05 s.

Explicitly unloading the co-resident models before the VAE makes the same workload stable at ~16 s.

### Suspected area

I do not know whether the best fix belongs in ComfyUI model-management policy or in comfy-aimdo's eviction/headroom behavior.

My current hypothesis is an interaction between:

- ComfyUI intentionally retaining one Dynamic VRAM model while another Dynamic VRAM model is used, relying on on-demand eviction;
- H3 weights remaining resident;
- VideoVAE requiring additional **non-weight** activation/workspace allocations during decode;
- reactive VBAR eviction/page faults occurring only once memory pressure is already critical.

This may create an eviction/page-in thrash loop near the VRAM limit. A small amount of proactive eviction/headroom before VAE decode may be much faster than maintaining maximum weight residency until allocations fault.

### Notes

- Sol attention / H3 Cache are not responsible for the measured variability: the isolated VAE-only repetitions do not re-run the sampler at all.
- `--disable-pinned-memory` was tested and does not eliminate the VAE timing variability.
- `--disable-async-offload` was already enabled in the test environment.
- This seems related in theme to #12927 and #13423. #14157 also reports intermittent VAE slowdown under insufficient VRAM headroom, although that report has a different NVIDIA/nvfp4-specific cause.
- I can provide the full `--verbose DEBUG` logs and the minimal diagnostic node/workflow if useful.

Start at the ComfyUI model-management entry points used by comfy.model_management.unload_all_models() and soft_empty_cache(), then compare normal Dynamic VRAM residency with the forced-unload diagnostic workflow. Reproduce the cached-latent VAE-only runs with verbose logging and inspect the related Dynamic VRAM/Aimdo eviction behavior. Done means repeated VideoVAE decodes avoid critical allocation pressure and remain close to the approximately 16-second forced-unload baseline.

Written by the indexing model from the issue text.