GPU Failure When Using Hunyuan Video with High Sampler Resolution in ComfyUI

### Expected Behavior

Hunyuan Video should be able to run stably when the sampler resolution is set above 512x512 and frames are set to 89 or higher, without causing the GPU to crash, affecting CUDA services, or forcing a system reboot.

### Actual Behavior

When using Hunyuan Video, if the sampler resolution exceeds 512x512 and frames are set to 89, the GPU is highly likely to crash, with the following symptoms:

When using remote access:

After the GPU failure, I can reconnect after a while.
However, CUDA services do not recover unless the system is rebooted.
When using the local machine:

The screen goes black, and the system becomes unresponsive.
A manual restart is required, or the system automatically reboots after some time.
Hardware monitoring shows normal temperatures and metrics, except that the GPU frequency is slightly higher than usual.

I have tested the latest driver and two previous versions, but the issue persists.

### Steps to Reproduce

Open ComfyUI and use Hunyuan Video.
Set the sampler resolution above 512x512 and frames to 89 or higher.
Start the video generation process.
During the rendering process, the GPU is highly likely to crash, leading to:
CUDA services becoming unusable (requiring a reboot).
Screen going black if using the local machine.
System instability, requiring a forced or automatic reboot.

### Debug Logs

[2025-01-04 13:16:01.829] Loading text encoder model (clipL) from: D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\models\clip\clip-vit-large-patch14
[2025-01-04 13:16:02.370] Text encoder to dtype: torch.float16
[2025-01-04 13:16:02.449] Loading tokenizer (clipL) from: D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\models\clip\clip-vit-large-patch14
[2025-01-04 13:16:02.570] Loading text encoder model (llm) from: D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\models\LLM\llava-llama-3-8b-text-encoder-tokenizer
[2025-01-04 13:16:17.370]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.00s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.65s/it]
[2025-01-04 13:16:19.610] Text encoder to dtype: torch.float16
[2025-01-04 13:16:21.824] Loading tokenizer (llm) from: D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\models\LLM\llava-llama-3-8b-text-encoder-tokenizer
[2025-01-04 13:16:22.226] D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\llama\modeling_llama.py:566: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.)
attn_output = torch.nn.functional.scaled_dot_product_attention(
[2025-01-04 13:16:22.260] llm prompt attention_mask shape: torch.Size([1, 161]), masked tokens: 9
[2025-01-04 13:16:25.149] clipL prompt attention_mask shape: torch.Size([1, 77]), masked tokens: 10
[2025-01-04 13:16:25.348] model_type FLOW
[2025-01-04 13:16:25.348] Scheduler config: FrozenDict([('num_train_timesteps', 1000), ('shift', 9.0), ('reverse', True), ('solver', 'euler'), ('n_tokens', None), ('_use_default_values', ['num_train_timesteps', 'n_tokens'])])
[2025-01-04 13:16:25.349] Using accelerate to load and assign model weights to device...
[2025-01-04 13:16:25.490] Loading LoRA: adapter_modelp3 with strength: 1.0
[2025-01-04 13:16:25.565] Requested to load HyVideoModel
[2025-01-04 13:16:34.854] loaded completely 9.5367431640625e+25 12555.953247070312 True
[2025-01-04 13:16:44.838] Input (height, width, video_length) = (960, 544, 129)
[2025-01-04 13:16:45.029] Swapping 20 double blocks and 0 single blocks
[2025-01-04 13:16:46.565] Sampling 129 frames in 33 latents at 544x960 with 39 inference steps
[2025-01-04 13:25:20.508]
[2025-01-04 13:25:20.514] Exception in thread Thread-11 (prompt_worker):
[2025-01-04 13:25:20.514] Traceback (most recent call last):
[2025-01-04 13:25:20.514] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\execution.py", line 328, in execute
[2025-01-04 13:25:20.521] output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
[2025-01-04 13:25:20.522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.527] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\execution.py", line 203, in get_output_data
[2025-01-04 13:25:20.527] return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
[2025-01-04 13:25:20.528] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.534] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\execution.py", line 174, in _map_node_over_list
[2025-01-04 13:25:20.534] process_inputs(input_dict, i)
[2025-01-04 13:25:20.534] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\execution.py", line 163, in process_inputs
[2025-01-04 13:25:20.534] results.append(getattr(obj, func)(**inputs))
[2025-01-04 13:25:20.534] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.536] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-HunyuanVideoWrapper\nodes.py", line 1143, in process
[2025-01-04 13:25:20.551] out_latents = model["pipe"](
[2025-01-04 13:25:20.552] ^^^^^^^^^^^^^^
[2025-01-04 13:25:20.553] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
[2025-01-04 13:25:20.562] return func(*args, **kwargs)
[2025-01-04 13:25:20.562] ^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.564] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-HunyuanVideoWrapper\hyvideo\diffusion\pipelines\pipeline_hunyuan_video.py", line 732, in __call__
[2025-01-04 13:25:20.571] noise_pred = self.transformer( # For an input image (129, 192, 336) (1, 256, 256)
[2025-01-04 13:25:20.571] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.575] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
[2025-01-04 13:25:20.586] return self._call_impl(*args, **kwargs)
[2025-01-04 13:25:20.587] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.590] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
[2025-01-04 13:25:20.590] return forward_call(*args, **kwargs)
[2025-01-04 13:25:20.590] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.593] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-HunyuanVideoWrapper\hyvideo\modules\models.py", line 978, in forward
[2025-01-04 13:25:20.600] x = block(*single_block_args)
[2025-01-04 13:25:20.602] ^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.603] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
[2025-01-04 13:25:20.604] return self._call_impl(*args, **kwargs)
[2025-01-04 13:25:20.604] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.606] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
[2025-01-04 13:25:20.606] return forward_call(*args, **kwargs)
[2025-01-04 13:25:20.606] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.607] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-HunyuanVideoWrapper\hyvideo\modules\models.py", line 454, in forward
[2025-01-04 13:25:20.608] attn = attention(
[2025-01-04 13:25:20.608] ^^^^^^^^^^
[2025-01-04 13:25:20.609] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-HunyuanVideoWrapper\hyvideo\modules\attention.py", line 162, in attention
[2025-01-04 13:25:20.615] x = sageattn_varlen_func(
[2025-01-04 13:25:20.615] ^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.616] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\_dynamo\eval_frame.py", line 451, in _fn
[2025-01-04 13:25:20.624] return fn(*args, **kwargs)
[2025-01-04 13:25:20.624] ^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.625] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-HunyuanVideoWrapper\hyvideo\modules\attention.py", line 23, in sageattn_varlen_func
[2025-01-04 13:25:20.625] return sageattn_varlen(q, k, v, cu_seqlens_q, cu_seqlens_kv, max_seqlen_q, max_seqlen_kv)
[2025-01-04 13:25:20.625] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.628] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\sageattention\core.py", line 198, in sageattn_varlen
[2025-01-04 13:25:20.633] q_int8, q_scale, k_int8, k_scale, cu_seqlens_q_scale, cu_seqlens_k_scale = per_block_int8_varlen(q, k, cu_seqlens_q, cu_seqlens_k, max_seqlen_q, max_seqlen_k, sm_scale=sm_scale)
[2025-01-04 13:25:20.633] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.638] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\sageattention\quant_per_block_varlen.py", line 62, in per_block_int8
[2025-01-04 13:25:20.643] q_scale = torch.empty((cu_seqlens_q_scale[-1], h_qo, 1), device=q.device, dtype=torch.float32)
[2025-01-04 13:25:20.644] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.650] RuntimeError: CUDA error: unknown error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[2025-01-04 13:25:20.651]
[2025-01-04 13:25:20.651]
[2025-01-04 13:25:20.651] During handling of the above exception, another exception occurred:
[2025-01-04 13:25:20.651]
[2025-01-04 13:25:20.651] Traceback (most recent call last):
[2025-01-04 13:25:20.651] File "threading.py", line 1045, in _bootstrap_inner
[2025-01-04 13:25:20.651] File "threading.py", line 982, in run
[2025-01-04 13:25:20.652] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\main.py", line 172, in prompt_worker
[2025-01-04 13:25:20.659] e.execute(item[2], prompt_id, item[3], item[4])
[2025-01-04 13:25:20.659] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\execution.py", line 505, in execute
[2025-01-04 13:25:20.659] result, error, ex = execute(self.server, dynamic_prompt, self.caches, node_id, extra_data, executed, prompt_id, execution_list, pending_subgraph_results)
[2025-01-04 13:25:20.659] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.666] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\execution.py", line 395, in execute
[2025-01-04 13:25:20.666] input_data_formatted[name] = [format_value(x) for x in inputs]
[2025-01-04 13:25:20.666] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.670] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\execution.py", line 395, in
[2025-01-04 13:25:20.670] input_data_formatted[name] = [format_value(x) for x in inputs]
[2025-01-04 13:25:20.670] ^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.672] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\ComfyUI\execution.py", line 247, in format_value
[2025-01-04 13:25:20.672] return str(x)
[2025-01-04 13:25:20.672] ^^^^^^
[2025-01-04 13:25:20.673] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\_tensor.py", line 464, in __repr__
[2025-01-04 13:25:20.684] return torch._tensor_str._str(self, tensor_contents=tensor_contents)
[2025-01-04 13:25:20.684] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.687] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\_tensor_str.py", line 697, in _str
[2025-01-04 13:25:20.696] return _str_intern(self, tensor_contents=tensor_contents)
[2025-01-04 13:25:20.696] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.698] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\_tensor_str.py", line 617, in _str_intern
[2025-01-04 13:25:20.698] tensor_str = _tensor_str(self, indent)
[2025-01-04 13:25:20.698] ^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-01-04 13:25:20.700] File "D:\newcom\ComfyUI_video\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\_tensor_str.py", line 331, in _tensor_str
[2025-01-04 13:25:20.700] self = self.float()
[2025-01-04 13:25:20.700] ^^^^^^^^^^^^
[2025-01-04 13:25:20.700] RuntimeError: CUDA error: unknown error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[2025-01-04 13:25:20.700]

### Other

_No response_