[release/2.14] 20 UT cases hang (>10 min timeout) on BMG: sparse / linalg / stft / multi_head_attention

Summary

In the 2.14 BMG (B60) unit-test run, a number of cases were reported by pytest-xdist as failed on setup with "worker 'gwN' crashed while running '<nodeid>'". Each named <nodeid> was re-run individually on BMG with a hard 11-minute (660s) cap; the following 20 cases still hang past 10 minutes (killed at 660s), i.e. they are genuine timeouts that take down the xdist worker and cause collateral setup failures for other cases on the same worker.

Environment

• torch 2.14.0.dev+xpu, BMG (B60), 8 GPUs
• PYTORCH_TEST_WITH_SLOW=1

Reproducer

cd pytorch/third_party/torch-xpu-ops/test/xpu
export PYTORCH_TEST_WITH_SLOW=1
ZE_AFFINITY_MASK=0 pytest <nodeid>   # hangs > 10 min

Verified timeout cases (killed at 660s)

Cases:
op_ut,third_party.torch-xpu-ops.test.xpu.test_indexing_xpu.TestIndexingXPU,test_index_add_fast_path_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_fn_fwgrad_bwgrad_stft_xpu_complex128
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_gradgrad_stft_xpu_complex128
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseXPU,test_index_select_empty_and_non_contiguous_index_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseXPU,test_to_sparse_xpu_complex128
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseAnyXPU,test_binary_operation_mul_SparseCSR_xpu_complex128
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseAnyXPU,test_binary_operation_mul_SparseBSC_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseAnyXPU,test_binary_operation_mul_SparseCSC_xpu_complex128
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseAnyXPU,test_binary_operation_mul_SparseBSC_xpu_complex128
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseAnyXPU,test_binary_operation_mul_SparseBSR_xpu_complex128
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseAnyXPU,test_binary_operation_mul_SparseCSR_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseXPU,test_index_select_exhaustive_index_large_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseXPU,test_to_sparse_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseAnyXPU,test_binary_operation_mul_SparseCSC_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseAnyXPU,test_gradcheck_to_dense_SparseCOO_int64_masked_xpu_complex128
op_extended,third_party.torch-xpu-ops.test.xpu.extended.test_ops_xpu.TestCompositeComplianceXPU,test_forward_ad_nn_functional_multi_head_attention_forward_xpu_float32
op_extended,third_party.torch-xpu-ops.test.xpu.extended.test_ops_xpu.TestCompositeComplianceXPU,test_operator_nn_functional_multi_head_attention_forward_xpu_float32
op_ut,third_party.torch-xpu-ops.test.xpu.test_sparse_xpu.TestSparseAnyXPU,test_binary_operation_mul_SparseBSR_xpu_float64

extended/test_ops_xpu.py — TestCompositeComplianceXPU

• test_forward_ad_nn_functional_multi_head_attention_forward_xpu_float32
• test_operator_nn_functional_multi_head_attention_forward_xpu_float32

test_indexing_xpu.py — TestIndexingXPU

• test_index_add_fast_path_xpu_float64

test_ops_fwd_gradients_xpu.py — TestFwdGradientsXPU

• test_fn_fwgrad_bwgrad_stft_xpu_complex128

test_ops_gradients_xpu.py — TestBwdGradientsXPU

• test_fn_gradgrad_stft_xpu_complex128

test_sparse_xpu.py — TestSparseAnyXPU

• test_binary_operation_mul_SparseBSC_xpu_complex128
• test_binary_operation_mul_SparseBSC_xpu_float64
• test_binary_operation_mul_SparseBSR_xpu_complex128
• test_binary_operation_mul_SparseCSC_xpu_complex128
• test_binary_operation_mul_SparseCSC_xpu_float64
• test_binary_operation_mul_SparseCSR_xpu_complex128
• test_binary_operation_mul_SparseCSR_xpu_float64

test_sparse_xpu.py — TestSparseXPU

• test_index_select_empty_and_non_contiguous_index_xpu_float64
• test_index_select_exhaustive_index_large_xpu_float64
• test_to_sparse_xpu_complex128
• test_to_sparse_xpu_float64

Notes

• The hangs are often dtype-specific: e.g. test_binary_operation_mul_SparseBSR_xpu_float64 and test_fn_gradgrad_stft_xpu_float64 complete quickly, while their complex128 variants hang. This points at slow/hanging complex128 (and some float64) paths in sparse mul, index_select/to_sparse, linalg low-rank/lobpcg, stft grad, and multi_head_attention forward.
• The other cases named in the same worker-crash messages were verified as not timeouts (they pass, fail fast, or crash for other reasons) and are excluded here.

Start with the listed tests in test_indexing_xpu.py, test_ops_fwd_gradients_xpu.py, test_ops_gradients_xpu.py, test_sparse_xpu.py, and extended/test_ops_xpu.py. Run each named node individually using the provided BMG command and compare the dtype-specific cases, especially complex128 and float64. Done means the verified cases no longer hang past the 660-second cap and no longer take down xdist workers.