Process abort (faulthandler stack at F.silu) with MiniMax H3 fp8_scaled - only when SageAttention + dynamic VRAM are both enabled at large sequence length

# Process abort (`Fatal Python error: Aborted`, faulthandler stack at `F.silu`) with MiniMax H3 fp8_scaled — only when SageAttention + dynamic VRAM are both enabled

## Summary

With the MiniMax H3 `ref2va_pruned_fp8_scaled` checkpoint, ComfyUI dies with a native
`Fatal Python error: Aborted` on the first denoising step — but **only when
`--use-sage-attention` and dynamic VRAM (default) are both active**. Removing either factor,
or serializing kernel launches, makes the identical graph run clean:

| Variation (same graph, same seed) | Result |
|---|---|
| SageAttention + dynamic VRAM (default) | ❌abort, 6/6 across fresh processes |
| SageAttention + dynamic VRAM + full machine reboot | ❌still aborts |
| SageAttention + dynamic VRAM + cleared triton cache | ❌still aborts |
| SageAttention + dynamic VRAM, **no custom nodes installed at all** (none imported) | ❌still aborts, same stack |
| SageAttention + dynamic VRAM + `CUDA_LAUNCH_BLOCKING=1` | ✅clean (95—01 s/it) |
| SageAttention + `--disable-dynamic-vram` | ✅clean (93.8 s/it; also a full 20-step run, 25.4 min) |
| **default PyTorch attention** (no `--use-sage-attention`) + dynamic VRAM | ✅clean |
| SageAttention + dynamic VRAM, same checkpoint, **smaller size** (1344×768 / 124 frames, ~38k tokens) | ✅clean |
| SageAttention + dynamic VRAM, int8_convrot checkpoint | ✅clean (many long runs) |
| SageAttention + dynamic VRAM, bf16 checkpoint | ✅clean |

This isolates the failure to an interaction involving the SageAttention path, dynamic VRAM,
asynchronous execution, and this fp8_scaled checkpoint path **at large sequence length**
(~169k packed tokens; ~38k is clean). A stream-synchronization or
tensor-lifetime issue is one possibility, but the current evidence does not identify the
failing kernel or root cause. Possibly related to the same subsystem as #15365 (fp8
non-determinism under dynamic VRAM); this failure mode is a hard process abort rather than
output noise.

The faulthandler stack is identical in all six crashes (note: this is where the Python
thread was when the process aborted, not necessarily where the fault originated):

```
Fatal Python error: Aborted

Stack (most recent call first):
File "...\torch\nn\functional.py", line 2371 in silu
File "...\ComfyUI\comfy\ops.py", line 959 in _swiglu_eager
File "...\ComfyUI\comfy\ops.py", line 982 in linear_input_act
File "...\ComfyUI\comfy\ldm\minimax\model.py", line 182 in forward
...
```

## Repro

- Model: `minimax_h3_ref2va_pruned_fp8_scaled.safetensors` (Comfy-Org repackage; local
SHA256 `f86f2f79ebd2d76eb8eeb46091e83982e6ff51d255747e7b16e92834b392b8e9` matches the hub)
- Graph: stock native H3 ref2va graph (UNETLoader →MiniMaxH3ReferenceToVideo with 3
reference images →BasicScheduler(res_multistep/beta) →BasicGuider → SamplerCustomAdvanced), seed fixed, **no custom nodes in the graph** (and one arm with no
custom nodes installed at all). API-format workflow JSON attached below.
- Size: 1664×960, 362 frames (~169k packed tokens). Crashes with steps=2 as well as
steps=20 — i.e. the failure is in the first denoising step, not accumulated over time.
**Sequence scale is part of the trigger**: the same three factors at 1344×768 / 124
frames (~38k tokens) run clean. We did not bisect the boundary between 38k and 169k
tokens; this may point at VRAM pressure / address reuse at large activation sizes rather
than the checkpoint format alone.
- Launch (failing): `python -s main.py --use-sage-attention`
- The crash occurs during/right after the "Model Initializing" phase of the first step.

Model-load log lines for the failing checkpoint (relevant because the checkpoint is
mixed-precision quantized — we have not verified at operator level which comfy_kitchen op
executes the failing linear, hence "fp8_scaled checkpoint path" rather than claims about a
specific fp8 kernel):

```
Found quantization metadata version 1
Detected mixed precision quantization
Using mixed precision operations
Native ops: int8_tensorwise, float8_e5m2, convrot_w4a4, nvfp4, float8_e4m3fn , emulated ops: mxfp8
model weight dtype torch.bfloat16, manual cast: torch.bfloat16
model_type FLOW_AV
```

## One historical inconsistency, disclosed

One full 20-step run with the same three factors active **did** complete about 30 hours
before the first crash. The ComfyUI checkout (commit below) and our Python environment did
not change in that window (no git pull, no package installs by us); OS/driver background
updates were not independently ruled out. So: the failure has not been universally
deterministic across all historical runs — compatible with, but not proof of, a timing- or
state-sensitive defect. In the current environment it is 6/6 reproducible.

## Environment

- ComfyUI commit `0dd9b154a1654fc699dcdc3af066c7cce096045a` (2026-08-06), tracked working
tree unmodified. Not the newest upstream — happy to retest on latest if requested.
- comfy-kitchen 0.2.26 (CUDA backend), **comfy-aimdo 0.4.11** — dynamic VRAM's CUDA-detour
component is active in the failing runs (`aimdo: installing 6 hooks`,
`comfy-aimdo NVML pressure enabled`). It is the pip-installed component shipped with this
ComfyUI install, not a third-party add-on; it stays active in the "no custom nodes" arm.
- sageattention 2.2.0+cu130torch2.9.0andhigher.post4
- torch 2.9.1+cu130, Python 3.12, Windows 11
- NVIDIA driver 610.47, RTX 5090 32 GB (sm_120), 64 GB system RAM

Full console log from startup to abort, and the API workflow JSON, attached. The machine
reproduces this reliably — happy to run additional diagnostics (operator dispatch logging,
a debug build, specific env flags).

## Attachments

API-format workflow JSON (sanitized repro)

```json
{
"1": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "minimax_h3_ref2va_pruned_fp8_scaled.safetensors",
"weight_dtype": "default"
}
},
"2": {
"class_type": "CLIPLoader",
"inputs": {
"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
"type": "minimax"
}
},
"3": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "minimax_h3_video_vae_fp16.safetensors"
}
},
"4": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "minimax_h3_audio_vae_fp32.safetensors"
}
},
"6": {
"class_type": "MiniMaxH3ReferenceToVideo",
"inputs": {
"clip": [
"2",
0
],
"vae": [
"3",
0
],
"audio_vae": [
"4",
0
],
"prompt": "A character dances through a cozy living room while a robot vacuum follows them around. Upbeat music, footsteps, vacuum hum.",
"width": 1664,
"height": 960,
"length": 362,
"ref_image_size": "match",
"ref_images.ref_image_0": [
"20",
0
],
"ref_images.ref_image_1": [
"21",
0
],
"ref_images.ref_image_2": [
"22",
0
]
}
},
"7": {
"class_type": "BasicScheduler",
"inputs": {
"model": [
"1",
0
],
"scheduler": "beta",
"steps": 2,
"denoise": 1.0
}
},
"8": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 77
}
},
"9": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "res_multistep"
}
},
"10": {
"class_type": "BasicGuider",
"inputs": {
"model": [
"1",
0
],
"conditioning": [
"6",
0
]
}
},
"11": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": [
"8",
0
],
"guider": [
"10",
0
],
"sampler": [
"9",
0
],
"sigmas": [
"7",
0
],
"latent_image": [
"6",
1
]
}
},
"12": {
"class_type": "VAEDecode",
"inputs": {
"samples": [
"11",
0
],
"vae": [
"3",
0
]
}
},
"13": {
"class_type": "VAEDecodeAudio",
"inputs": {
"samples": [
"11",
0
],
"vae": [
"4",
0
]
}
},
"14": {
"class_type": "CreateVideo",
"inputs": {
"images": [
"12",
0
],
"audio": [
"13",
0
],
"fps": 24.0
}
},
"15": {
"class_type": "SaveVideo",
"inputs": {
"video": [
"14",
0
],
"filename_prefix": "video/h3_fp8_repro",
"format": "mp4",
"codec": "h264",
"codec.encoding": "re-encode",
"codec.encoding.crf": 15.0
}
},
"20": {
"class_type": "LoadImage",
"inputs": {
"image": "ref_character.png"
}
},
"21": {
"class_type": "LoadImage",
"inputs": {
"image": "ref_background.png"
}
},
"22": {
"class_type": "LoadImage",
"inputs": {
"image": "ref_prop.png"
}
}
}

```

Console log: startup environment + crash context (from the no-custom-nodes arm; full 450-line log available on request)

```text
=== startup ===
NFO] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_embedding', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
NFO] Found comfy_kitchen backend hip: {'available': False, 'disabled': False, 'unavailable_reason': 'PyTorch ROCm/HIP runtime not available', 'capabilities': []}
NFO] Found comfy_kitchen backend cuda: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
NFO] Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_rowwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_']}
NFO] Total VRAM 32607 MB, total RAM 63092 MB
NFO] pytorch version: 2.9.1+cu130
NFO] Set vram state to: NORMAL_VRAM
NFO] Device: cuda:0 NVIDIA GeForce RTX 5090 : cudaMallocAsync
NFO] Using sage attention
aimdo: src-win/cuda-detour.c:38:INFO:aimdo_setup_hooks: installing 6 hooks
aimdo: src-win/cuda-detour.c:28:DEBUG:install_hook_entries: hooks successfully installed
aimdo: src/control.c:262:INFO:comfy-aimdo NVML pressure enabled
aimdo: src-win/shmem-detect.c:80:INFO:comfy-aimdo WDDM adapter match: NVIDIA GeForce RTX 5090 runtime_luid=00000000:0001a976 dxgi_luid=00000000:0001a976
aimdo: src/control.c:277:INFO:comfy-aimdo inited for GPU: NVIDIA GeForce RTX 5090 (VRAM: 32606 MB)
NFO] Python version: 3.13.11 (tags/v3.13.11:6278944, Dec 5 2025, 16:26:58) [MSC v.1944 64 bit (AMD64)]
NFO] ComfyUI version: 0.30.0
NFO] comfy-aimdo version: 0.4.11
Installed comfy-aimdo version 0.4.11 is lower than the recommended version 0.4.13.
NFO] Total VRAM 32607 MB, total RAM 63092 MB
NFO] pytorch version: 2.9.1+cu130
NFO] Set vram state to: NORMAL_VRAM
NFO] Device: cuda:0 NVIDIA GeForce RTX 5090 : cudaMallocAsync
NFO] ### ComfyUI Version: v0.30.0-21-g0dd9b154 | Released on '2026-08-06'
NFO] Using sage attention
Installed comfy-aimdo version 0.4.11 is lower than the recommended version 0.4.13.
NFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float32
NFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float16
NFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16

=== last lines before and after the abort ===
NFO] Using MixedPrecisionOps for text encoder
NFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
NFO] Requested to load MiniMaxH3VideoVAE
NFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.
NFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.
NFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.
NFO] Requested to load MiniMaxH3TEModel_
NFO] Model MiniMaxH3TEModel_ prepared for dynamic VRAM loading. 14956MB Staged. 0 patches attached. Force pre-loaded 410 weights: 4572 KB.
NFO] Found quantization metadata version 1
NFO] Detected mixed precision quantization
NFO] Using mixed precision operations
NFO] Native ops: float8_e4m3fn, float8_e5m2, int8_tensorwise, convrot_w4a4, nvfp4 , emulated ops: mxfp8
NFO] model weight dtype torch.bfloat16, manual cast: torch.bfloat16
NFO] model_type FLOW_AV
NFO] Requested to load MiniMaxH3
NFO] 0 models unloaded.
NFO] Model MiniMaxH3 prepared for dynamic VRAM loading. 19983MB Staged. 0 patches attached. Force pre-loaded 360 weights: 1175 KB.

0%| | 0/2 [00:00

Run the attached API workflow with --use-sage-attention under dynamic VRAM, then compare it with CUDA_LAUNCH_BLOCKING=1 and --disable-dynamic-vram. Read comfy/ops.py around _swiglu_eager and linear_input_act, and comfy/ldm/minimax/model.py around forward; done means identifying and correcting the abort while preserving the clean behavior of the comparison cases.

Written by the indexing model from the issue text.