GPU error encountered while running sglang/test_wave_attention_backend.py

This causes a GPU error when using cuda_graph with large input sizes. By default, SGLang’s unit tests include test_mmlu, which runs multi-task models for performance comparison. The test crashes during cuda_graph loading due to the large size.

python3 -m unittest -v test_wave_attention_backend.TestWaveAttnBackend.test_mmlu

""" Error message """
Memory[S, BH, N, K2].of(bf16))
[2025-07-25 01:57:23] Invoke IREE Tool: /usr/local/lib/python3.12/dist-packages/iree/compiler/tools/../_mlir_libs/iree-compile - --iree-input-type=auto --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=rocm --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false --iree-hal-target-backends=rocm --iree-vm-bytecode-module-strip-source-map=true --iree-opt-strip-assertions=true --iree-vm-target-truncate-unsupported-floats --iree-hip-target=gfx942 --iree-hal-dump-executable-binaries-to=/tmp/tmpcjszts8p
[2025-07-25 01:57:23] in_thread_transpose
[2025-07-25 01:57:23] Invoke IREE Tool: /usr/local/lib/python3.12/dist-packages/iree/compiler/tools/../_mlir_libs/iree-compile - --iree-input-type=auto --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=rocm --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false --iree-hal-target-backends=rocm --iree-vm-bytecode-module-strip-source-map=true --iree-opt-strip-assertions=true --iree-vm-target-truncate-unsupported-floats --iree-hip-target=gfx942 --iree-hal-dump-executable-binaries-to=/tmp/tmpcjszts8p
**Capturing batches (bs=384 avail_mem=26.66 GB): 16%|██▉ | 8/51 [00:08<00:17, 2.44it/s]Memory access fault by GPU node-2 (Agent handle: 0x14c29630) on address 0x7eb55e213000. Reason: Unknown.**
[2025-07-25 02:06:47] Child process unexpectedly failed with exitcode=9. pid=2690276
ERROR