[MiniMax H3][Linux][multi-GPU] reproducible whole-host hard resets with INT8 ConvRot while Wan 2.2 is stable

### Custom Node Testing

- [ ] I have tried disabling custom nodes and the issue persists

Important qualification: the first H3 hard resets occurred with ComfyUI's native CPU offload before the custom block-swap/instrumentation node used in later diagnosis existed. Later runs use a small H3-only node to control block residency and, most recently, to make the official SageAttention facade fail closed instead of silently falling back to PyTorch. The host-reset symptom is therefore not introduced by that node, but I have not yet reproduced the current exact workflow under `--disable-all-custom-nodes`.

### Expected Behavior

Five independent ComfyUI processes, each pinned to one local NVIDIA GPU with `CUDA_VISIBLE_DEVICES`, should either finish their MiniMax H3 jobs or fail in user space with a CUDA OOM / Python exception. A model workload must not instantaneously reset the entire Linux host without a logged shutdown, kernel panic, NVIDIA Xid, PCIe AER event, or OOM kill.

### Actual Behavior

MiniMax H3 FL2VA inference reproducibly causes a power-cycle-like reset of the entire Linux host under specific multi-GPU concurrency. This has happened more than ten times during controlled experiments between 2026-08-14 and 2026-08-20.

The persistent journal ends abruptly. There is no normal shutdown sequence. A remote collector receives normal 20 Hz system telemetry until the machine disappears; the machine then boots again. Immediately before the latest reset:

- all five GPUs were still visible and had active CUDA contexts;
- GPU utilization was 99-100% on all five cards;
- VRAM usage was 7,685 / 11,653 / 7,561 / 7,561 / 7,561 MiB;
- per-worker denoising progress was 12/20, 5/20, 3/20, 3/20, and 2/20 steps;
- total instantaneous GPU power was approximately 695 W, below a separately tested stable synthetic peak of 1,007 W;
- maximum GPU temperature was 82 C, with no hardware thermal slowdown;
- 60.5 GiB host RAM remained available and swap usage was zero;
- `oom_kill=0`, memory PSI was zero, and all monitored PCIe AER counters were zero.

The last remote sample was at `2026-08-20 20:30:20.620 CEST`; the next boot began at `20:30:54 CEST`. There is no exception or final error in the ComfyUI logs because the whole host disappears during sampling.

This is H3-path-specific on this host. A production Wan 2.2 batch completed 1,764 items without this failure. A strict Wan 2.2 control on the same five GPUs, same physical topology, same kernel/driver/BIOS boot, and zero dispatch stagger kept all five local cards active for 5 minutes 24 seconds and produced all outputs.

### Steps to Reproduce

#### Workload

1. Start five isolated ComfyUI processes, one per GPU, using `CUDA_VISIBLE_DEVICES=0` through `4` and separate ports/databases.
2. Start each process with:

```text
--use-sage-attention
--disable-dynamic-vram
--disable-pinned-memory
--reserve-vram 0.4
--cache-none
```

3. Use the Comfy-Org MiniMax H3 files:

```text
diffusion_models/minimax_h3_fl2va_pruned_int8_convrot.safetensors
text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors
vae/minimax_h3_video_vae_fp16.safetensors
vae/minimax_h3_audio_vae_fp32.safetensors
```

The diffusion checkpoint is the 20,970,379,616-byte Comfy-Org INT8 ConvRot **pruned repack**, not the original MiniMax BF16 checkpoint. This distinction is important: this report does not yet demonstrate a defect in the original BF16 weights.

4. Run FL2VA image-to-video at 864x480, 124 frames, video+audio joint diffusion, Euler/simple, 20 steps. Use any valid source image and prompt.
5. For 8 GiB cards, keep eight of 50 DiT blocks resident and swap 42 blocks. For the 12 GiB card, keep 19 blocks resident and swap 31. These values leave sufficient workspace for SageAttention; one fewer swapped block causes Sage OOM and ComfyUI's normal Sage wrapper silently falls back to PyTorch.
6. Dispatch to the five processes at `0/5/10/15/20` seconds.

The latest run used an H3-only fail-closed binding around ComfyUI's already imported SageAttention 2.2.0 facade. It preserves the facade's Ampere/sm86 Triton dispatch and default `smooth_k=True`, counts successful calls, and re-raises any Sage exception instead of using the PyTorch fallback. A 6-step validation produced 300 successful Sage calls per card, zero failures, and zero fallbacks. The otherwise identical 20-step run reset the host while all five cards were denoising.

The same host-reset symptom also predates this fail-closed binding and predates the block-swap node:

- default ComfyUI CPU offload + PyTorch SDPA: reset;
- custom block swap + PyTorch SDPA: reset;
- custom block swap + direct strict Sage: reset;
- official Sage facade, `smooth_k=True`, 20-step production run: reset;
- fail-closed official Sage, 20-step tight five-card run: reset.

#### Observed topology boundary

The host has one CPU-direct GPU and four PCH-side GPUs:

| Logical GPU | Model | PCIe path | Link |
|---|---|---|---|
| GPU 0 | RTX 3070 8 GiB | CPU root port, `01:00.0` | Gen3 x16 |
| GPU 1 | RTX 3060 12 GiB | PCH, `03:00.0` | Gen3 x4 |
| GPU 2 | RTX 3060 Ti 8 GiB | PCH, `05:00.0` | Gen3 x1 riser |
| GPU 3 | RTX 3060 Ti 8 GiB | PCH, `07:00.0` | Gen3 x1 riser |
| GPU 4 | RTX 3060 Ti 8 GiB | PCH, `08:00.0` | Gen3 x1 riser |

For the original PyTorch/block-swap runs, the observed fatal topology was GPU 0 active together with at least three PCH GPUs:

| Configuration | Result |
|---|---|
| GPU 0 + two PCH GPUs | >=1,537 s concurrent model residency, stable |
| four PCH GPUs without GPU 0 | >=972 s, stable |
| GPU 0 + three PCH GPUs, two different combinations | hard reset at 9m13s and 19m32s |
| all five local GPUs | repeated hard resets |

This topology condition is not sufficient by itself: the same five-card topology is stable with Wan 2.2, and a strict H3 6-step Sage run also succeeds. The failure is sensitive to the H3 execution phase, duration, and concurrency.

### Debug Logs

There is no software crash tail. The latest five worker logs end only in progress bars:

```text
GPU 0: 60% | 12/20 [02:49<01:53, 14.17s/it]
GPU 1: 25% | 5/20 [02:14<06:50, 27.36s/it]
GPU 2: 15% | 3/20 [01:43<09:49, 34.68s/it]
GPU 3: 15% | 3/20 [01:45<09:58, 35.20s/it]
GPU 4: 10% | 2/20 [01:09<10:23, 34.62s/it]
```

The last remote telemetry sample contains:

```text
load average: 7.23 / 4.96 / 4.10
MemAvailable: 63,480,184 KiB
SwapFree: 44,630,012 KiB (all swap free)
oom_kill: 0
memory PSI: avg10=0.0, avg60=0.0, avg300=0.0
PCIe AER: all monitored correctable/nonfatal/fatal counters = 0

GPU 0: util 100%, memory util 67%, 7685 MiB, 261.68 W, 82 C, Gen3 x16
GPU 1: util 99%, memory util 0%, 11653 MiB, 108.86 W, 62 C, Gen3 x4
GPU 2: util 100%, memory util 6%, 7561 MiB, 116.36 W, 63 C, Gen3 x1
GPU 3: util 100%, memory util 0%, 7561 MiB, 95.88 W, 57 C, Gen3 x1
GPU 4: util 100%, memory util 43%, 7561 MiB, 112.46 W, 56 C, Gen3 x1
hardware thermal slowdown: 0 on all cards
power brake: not active on all cards
```

Across fatal boots, searches of the persistent journal found no NVIDIA Xid, PCIe AER, MCE, kernel panic, OOM killer, thermal shutdown, link downgrade, or normal shutdown sequence.

### Other

#### Software environment

```text
OS: Ubuntu 22.04.5 LTS, x86_64
Kernel: 6.8.0-136-generic
NVIDIA module: NVIDIA Open Kernel Module 595.71.05
Python: 3.12.11
PyTorch: 2.11.0+cu130
Triton: 3.6.0
SageAttention: 2.2.0
comfy-kitchen: 0.2.31
comfy-aimdo: 0.4.13
ComfyUI: 0.33.0
ComfyUI commit: b963f4ad210a42841ab23dfc28a84143a0cce227
```

The reset also reproduced after these rollbacks/changes:

- kernel `6.8.0-136` -> `6.8.0-124`;
- NVIDIA open driver `595.84` -> `595.71.05`;
- VT-d/IOMMU disabled and enabled;
- pageable offload and a true `cudaHostRegister` experiment with 49.78 GiB pinned;
- default ComfyUI copy/offload, synchronous block swap, and stream+event copy without prefetch;
- PyTorch SDPA, direct Sage, and the official Sage facade.

#### Host hardware

```text
CPU: Intel Core i3-10100, 4 cores / 8 threads
RAM: 125.6 GiB, non-ECC
Motherboard: ASUS PRIME Z490-P
BIOS: American Megatrends 1410 (2020-09-04)
Storage: Samsung 970 EVO Plus 2 TB, PCH Gen3 x4
PSU: 1200 W high-quality unit
GPUs: RTX 3070 8 GiB + RTX 3060 12 GiB + 3x RTX 3060 Ti 8 GiB
```

#### Controls and eliminated simple explanations

| Test | Result |
|---|---|
| Wan 2.2 historical production | 1,764 completed items, stable |
| Wan 2.2 strict five-card zero-stagger control | 5m24s and all outputs, stable |
| Five-GPU synthetic power waveform | 1,007 W peak for 20 min, stable |
| Three 3060 Ti cards limited to 100 W | H3 still reset |
| Synthetic PCIe H2D + NVMe/DMI stress | exceeded fatal-run bandwidth, stable |
| H3-like duty-cycle synthetic load | 21 min, stable |
| VRAM occupancy + memory-bandwidth synthetic load | 12 min, stable |
| VCCSA/VCCIO +0.1 V | H3 still reset |
| Reduced H3: 6 steps, 576x736x56, compute dose approximately Wan | reset after 10 completed videos |
| Same reduced H3 workflow with SageAttention | 30m26s, 48 completed + 5 running, no fallback |
| Production 864x480x124, strict Sage, 6 steps, tight five-card dispatch | 5/5 completed, no fallback |
| Same strict Sage production workload, 20 steps | whole-host reset |

The reduced PyTorch-vs-Sage A/B had similar throughput and load. Sage was not simply lighter: GPU-utilization p95 was 100% in both; memory-utilization p95 increased from 55% to 64% with Sage; power p95 was 770.2 W vs 776.4 W; maximum temperature was 81 C in both.

#### Current interpretation and questions

The evidence does not support generic five-GPU instability, a simple PSU/temperature/RAM exhaustion problem, or the custom block-swap implementation as the cause. It points to an H3-specific interaction among the INT8 ConvRot repack, full-attention/other fused kernels, CPU offload/copy phases, CUDA driver, and multi-process Ampere execution. The exact failing component is still unknown.

Could the ComfyUI maintainers advise or provide:

1. a reference/eager path that disables the H3 INT8 ConvRot fused kernels independently of attention;
2. an H3 operator-level trace/debug switch to identify the last submitted kernel or copy before host loss;
3. confirmation of whether multi-process H3 INT8 ConvRot on Ampere is a tested configuration;
4. any known constraints in `comfy-kitchen` 0.2.31 for this checkpoint/topology;
5. a way to select SageAttention fail-closed, since the current wrapper catches every Sage exception and silently invokes PyTorch attention;
6. guidance on the most useful minimal follow-up: full non-pruned INT8 ConvRot, original BF16, single process with multiple devices, or a specific unfused kernel combination.

I can provide the sanitized workflows, complete startup/sampling logs, 20 Hz remote telemetry, per-boot journals, and the experiment matrix. Raw files contain local paths/host identifiers, so they are not attached publicly in this first report.

Related reports which may share part of the failure family:

- #15271 — illegal memory access; a report persists with Dynamic VRAM and Sage disabled
- #15360 — complete dual-GPU system freeze requiring a hard power cycle
- #15488 — H3 `GPU is lost` / black-screen, including multi-instance reports, while Wan is stable
- #15529 — H3 INT8 ConvRot + comfy-kitchen attention alignment failure
- #15566 — H3 FP8 + Sage/DynamicVRAM abort

The payload names no source file or test; start by reproducing the five-process MiniMax H3 workflow with the listed flags, checkpoint, topology, and dispatch timing. Compare the 6-step successful run with the 20-step reset while collecting the persistent journal, NVIDIA, PCIe, and worker logs. Done means identifying a specific reproducible software component or narrowing the failure to a documented execution path.

Written by the indexing model from the issue text.