MiniMax H3 reproducibly causes `GPU is lost` / TDR black-screen on RTX 5070 Ti when the system has 64 GB RAM; identical workload is stable when Windows is capped to 32 GB

## Summary

On a single-GPU Windows box, MiniMax H3 video generation reliably kills the GPU (`nvidia-smi: GPU is lost`, requires reboot) after 1–8 generations. The same machine, same GPU, same driver, same ComfyUI build is **completely stable** in two control conditions:

- **Cap Windows to 32 GB** (`bcdedit /set removememory 32768`, both DIMMs physically installed, XMP on): **29 consecutive H3 generations, zero incidents.**
- **Run a comparable non-comfy_kitchen model at full 64 GB** (Wan 2.1 14B fp16, 31.3 GB DiT, same DynamicVRAM streaming pressure, same ~300 s/run, same back-to-back queueing): **10 consecutive generations, zero incidents.**

So the failure appears to require (a) H3 / comfy_kitchen quantized inference **and** (b) the full 64 GB being visible to the OS. Neither alone reproduces it.

## Environment

| | |
|---|---|
| GPU | RTX 5070 Ti (Blackwell, 16 GB) |
| Driver | 610.47 **and** 610.88 — both affected |
| OS | Windows 11 Enterprise 26200 |
| CPU / RAM | i5-12400F / 2×32 GB DDR4, tested at 3200 and 2666 |
| ComfyUI | 0.30.1 **and** 0.31.1 — both affected |
| comfy-kitchen | 0.2.26 **and** 0.2.28 — both affected |
| comfy-aimdo | 0.4.11 **and** 0.4.13 — both affected |
| torch | 2.9.1+cu130 |
| Model | `minimax_h3_fl2va_int8_convrot` (31.7 GB) + `qwen3vl_32b_minimax_h3_nvfp4_awq` TE; DynamicVRAM streaming (model does not fit in 16 GB VRAM) |

## Symptom

Mid-sampling the GPU disappears from the bus:

```
nvidia-smi: Unable to determine the device handle for GPU0: 0000:01:00.0: GPU is lost.
Reboot the system to recover this GPU
```

Windows WER records, every time:

```
LKD_0x141_Tdr:6_IMAGE_nvlddmkm.sys_Blackwell
LKD_0x1B8_NV_Blackscreen_Blackbox_dxgkrnl!DxgCreateLiveDumpWithDriverBlob
```

No WHEA errors, no minidump. The OS survives (a scripted `shutdown /r` works), only the display driver is dead.

## Telemetry at the moment of failure

10-second-resolution sampling across 8 separate crashes shows **nothing anomalous** right up to the last sample:

```
GPU temp 74–88 °C (often falling just before the crash)
GPU power 234–280 W (TDP 300 W, never pegged)
GPU util 100 %, dropping to 58–80 % in the final sample only
VRAM 10.5–15.9 GB (never OOM)
Free system RAM 11–36 GB free (never under pressure)
Pagefile peak 0–37 MB (essentially untouched)
```

Memory pressure, thermals and power are all ruled out by direct measurement.

## Python traceback is not a reliable fault location

The reported location differs every time, and in one case it is `torch.ones(())` — an operation that cannot itself hang a GPU. This is consistent with the async-error warning ComfyUI already prints; listing them only to show the spread:

```
# ComfyUI 0.30.1
comfy/memory_management.py:32 read_tensor_file_slice_into
destination2._params.copy_from(..., non_blocking=True)
-> comfy_kitchen/tensor/base.py:86 copy_from

# ComfyUI 0.31.1, cuda backend
comfy_kitchen/backends/cuda/__init__.py:1586 quantize_and_rotate_rowwise
-> comfy_kitchen/tensor/int8_utils.py:72 _rotate_activation
x_rotated = torch.matmul(x_grouped, h)

# ComfyUI 0.31.1, triton backend forced first
comfy_kitchen/backends/triton/quantization.py:1055 int8_linear
-> comfy_kitchen/tensor/int8_utils.py:72 _rotate_activation (same shared helper)

# ComfyUI 0.31.1, nvfp4 weights
comfy/quant_ops.py:132 quantize
scale = torch.ones((), device=tensor.device, dtype=torch.float32)
```

All four end in `torch.AcceleratorError: CUDA error: unknown error`.

## Ruled out by experiment (each tested to the point of a crash or a clean run set)

- NVIDIA driver: 610.47 and 610.88 both crash
- ComfyUI: 0.30.1 and 0.31.1 both crash
- comfy-kitchen 0.2.26→0.2.28, comfy-aimdo 0.4.11→0.4.13: both crash
- `--disable-pinned-memory`: still crashes (run 6)
- `--disable-async-offload` (NUM_STREAMS=0, prefetch queue removed): still crashes (run 6)
- comfy_kitchen backend: cuda **and** triton (forced via `set_backend_priority(["triton","cuda","eager"])`) both crash
- Quantization: `int8_convrot`, `pruned_int8_convrot`, and a third-party `nvfp4` build all crash
- Sampler steps 4 / 6 / 8; samplers `dual_clock_euler` and `er_sde`; 0.4 MP and 0.7 MP — all crash
- RAM: MemTest86 v11.7, 4/4 passes + row-hammer over the full 68.5 GB at 3192 MT/s — **PASS, 0 errors**
- RAM speed: XMP off (DDR4-2666) still crashes (run 15)
- Model size / staging volume: pruned build (20.0 GB staged, 24 GB RAM free at crash) crashes too
- Thermals, power, PSU, motherboard, GPU hardware: same hardware ran 41 consecutive clean H3 generations before the RAM upgrade

## The two variables that do change the outcome

1. **RAM visible to Windows.** 32 GB visible → 29 clean runs. 64 GB visible → crashes within 1–8 runs, across every software configuration above.
2. **Whether the model goes through comfy_kitchen quantized inference.** Wan 2.1 fp16 at 64 GB: 10 clean runs under equivalent streaming load.

For reference, the measured difference between the two RAM conditions is that at 32 GB the 31.7 GB weight file is re-read from disk every step (sustained 600–976 MB/s), whereas at 64 GB it is served from the OS page cache (0 MB/s disk). ComfyUI-side constants that scale with total RAM are `cache_ram = min(10, max(2, total_ram*0.10/1024))`, `cache_ram_inactive = min(128, total_ram/1024)` (main.py) and `MAX_PINNED_MEMORY = ram * 0.40` (model_management.py).

## Repro

1. 16 GB VRAM Blackwell card, ≥64 GB system RAM, Windows.
2. Load `minimax_h3_fl2va_int8_convrot` + `qwen3vl_32b_minimax_h3_nvfp4_awq` (DynamicVRAM engages; model does not fit VRAM).
3. Queue 10+ generations at 0.7 MP / 243 frames back to back without restarting ComfyUI.
4. GPU is lost within 1–8 runs.
5. `bcdedit /set removememory 32768` + reboot → same queue completes 29/29.

## Question

Is there a known path by which the DynamicVRAM / comfy_kitchen quantized path can wedge the GPU, and is there a supported way to make ComfyUI behave as if the machine had less RAM (i.e. pin `cache_ram` / `MAX_PINNED_MEMORY` to the smaller values) without physically hiding memory from the OS?

I have 8 crash captures with 10-second telemetry, the full control-experiment matrix, and Windows event logs, and can supply any subset on request.

Reproduce the H3 workload under 32 GB and 64 GB visible RAM, then inspect the RAM-dependent settings in main.py and model_management.py, especially cache_ram, cache_ram_inactive, and MAX_PINNED_MEMORY. Trace the quantized DynamicVRAM paths mentioned in comfy/memory_management.py and comfy_kitchen/backends/cuda/__init__.py, comfy_kitchen/backends/triton/quantization.py, and comfy_kitchen/tensor/int8_utils.py. Done means identifying a supported configuration or a confirmed bug with a narrowly defined fix and verification against the reported controls.

Written by the indexing model from the issue text.