[B60][XPU] Large tensor tests newly fail/skip with insufficient XPU memory vs PVC
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 107
Description
## Summary
B60 reports multiple XPU memory-capacity failures/skips while the same rows passed on PVC.
## Failure Category
Insufficient XPU memory for large tensor / 64-bit indexing tests.
Representative messages:
```text
Insufficient xpu memory
Insufficient xpu:0 memory
insufficient memory
```
## Scope
- Test suites: stock PyTorch and torch-xpu-ops
- Platform comparison: B60 skipped/failed; PVC passed
- Affected rows observed: 25
## Representative Stock PyTorch Rows
```text
test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_conv3d_xpu
Skipped: insufficient memory
test/inductor/test_mix_order_reduction.py::MixOrderReductionTest::test_out_of_shared_memory
Skipped: Insufficient xpu memory
test/nn/test_convolution.py::TestConvolutionNNDeviceTypeXPU::test_conv3d_64bit_indexing_xpu
Skipped: Insufficient xpu:0 memory
```
## Representative torch-xpu-ops Rows
Direct torch-xpu-ops XPU test files:
```text
test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_warp_softmax_64bit_indexing_xpu_float32
test/xpu/nn/test_convolution_xpu.py::TestConvolutionNNDeviceTypeXPU::test_conv3d_64bit_indexing_xpu
test/xpu/nn/test_convolution_xpu.py::TestConvolutionNNDeviceTypeXPU::test_conv_large_batch_1_xpu
test/xpu/test_foreach_xpu.py::TestForeachXPU::test_foreach_copy_with_multi_dtypes_large_input_xpu
test/xpu/test_linalg_xpu.py::TestLinalgXPU::test_triu_tril_large_matrix_64bit_xpu
test/xpu/test_sparse_xpu.py::TestSparseXPU::test_coalesce_accepts_large_tensor_xpu_float32
test/xpu/test_tensor_creation_ops_xpu.py::TestRandomTensorCreationXPU::test_randperm_large_xpu
test/xpu/test_torch_xpu.py::TestTorchDeviceTypeXPU::test_masked_scatter_large_tensor_xpu
```
## Expected Behavior
These rows should not newly skip/fail for insufficient XPU memory on B60 when PVC passes.
## Notes
This may be environment/device-memory capacity rather than operator correctness, but it is tracked separately because the message family is consistent and affects many large-tensor and 64-bit indexing tests.
## Full Case List
Total cases: 25
Show all 25 cases
1. `stock pytorch` | `test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_conv3d_xpu` | `skipped`
- `insufficient memory`
2. `stock pytorch` | `test/inductor/test_mix_order_reduction.py::MixOrderReductionTest::test_out_of_shared_memory` | `skipped`
- `Insufficient xpu memory`
3. `stock pytorch` | `test/nn/test_convolution.py::TestConvolutionNNDeviceTypeXPU::test_conv3d_64bit_indexing_xpu` | `skipped`
- `Insufficient xpu:0 memory`
4. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_warp_softmax_64bit_indexing_xpu_float32` | `failed`
- `Insufficient xpu memory`
5. `torch-xpu-ops` | `test/xpu/nn/test_convolution_xpu.py::TestConvolutionNNDeviceTypeXPU::test_conv3d_64bit_indexing_xpu` | `skipped`
- `Insufficient xpu memory`
6. `torch-xpu-ops` | `test/xpu/nn/test_convolution_xpu.py::TestConvolutionNNDeviceTypeXPU::test_conv_large_batch_1_xpu` | `skipped`
- `Insufficient xpu memory`
7. `torch-xpu-ops` | `test/xpu/test_foreach_xpu.py::TestForeachXPU::test_foreach_copy_with_multi_dtypes_large_input_xpu` | `skipped`
- `Insufficient xpu memory`
8. `torch-xpu-ops` | `test/xpu/test_linalg_xpu.py::TestLinalgXPU::test_triu_tril_large_matrix_64bit_xpu` | `skipped`
- `Insufficient xpu memory`
9. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_avg_pool_large_tensor2_xpu` | `skipped`
- `Insufficient xpu memory`
10. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_large_max_pool_contig_xpu` | `skipped`
- `Insufficient xpu memory`
11. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_nll_loss_large_tensor_reduction_mean_xpu` | `skipped`
- `Insufficient xpu memory`
12. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_nll_loss_large_tensor_reduction_none_xpu` | `skipped`
- `Insufficient xpu memory`
13. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_nll_loss_large_tensor_reduction_sum_xpu` | `skipped`
- `Insufficient xpu memory`
14. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_softmax_forward_64bit_indexing_xpu` | `skipped`
- `Insufficient xpu memory`
15. `torch-xpu-ops` | `test/xpu/test_sparse_xpu.py::TestSparseXPU::test_coalesce_accepts_large_tensor_xpu_float32` | `skipped`
- `Insufficient xpu memory`
16. `torch-xpu-ops` | `test/xpu/test_tensor_creation_ops_xpu.py::TestRandomTensorCreationXPU::test_randperm_large_xpu` | `skipped`
- `Insufficient xpu memory`
17. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNN::test_layer_norm_large_tensor` | `skipped`
- `Insufficient xpu memory`
18. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_replicatepad_64bit_indexing_xpu_float16` | `skipped`
- `Insufficient xpu:0 memory`
19. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_upsampling_64bit_indexing_channels_last_xpu_bfloat16` | `skipped`
- `Insufficient xpu:0 memory`
20. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_upsampling_64bit_indexing_channels_last_xpu_float16` | `skipped`
- `Insufficient xpu:0 memory`
21. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_upsamplingnearest2d_backward_64bit_indexing_xpu_float16` | `skipped`
- `Insufficient xpu:0 memory`
22. `torch-xpu-ops` | `test/xpu/test_tensor_creation_ops_xpu.py::TestLikeTensorCreationXPU::test_ones_large_xpu_complex128` | `skipped`
- `Insufficient xpu:0 memory`
23. `torch-xpu-ops` | `test/xpu/test_tensor_creation_ops_xpu.py::TestLikeTensorCreationXPU::test_zeros_large_xpu_complex128` | `skipped`
- `Insufficient xpu:0 memory`
24. `torch-xpu-ops` | `test/xpu/test_torch_xpu.py::TestTorchDeviceTypeXPU::test_cumsum_64bit_indexing_xpu` | `skipped`
- `Insufficient xpu:0 memory`
25. `torch-xpu-ops` | `test/xpu/test_torch_xpu.py::TestTorchDeviceTypeXPU::test_masked_scatter_large_tensor_xpu` | `skipped`
- `Insufficient xpu:0 memory`
## Reproducer / Environment Setup
```bash
docker run -it -e TZ=Asia/Shanghai --device=/dev/mem --device=/dev/dri --group-add video --privileged -v $(realpath ${HOME}):/home/jenkins --shm-size=8g intelgpu/ubuntu-26.04-rolling:26.18 bash
apt update
apt upgrade -y
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv myvenv --python 3.10 --clear
source myvenv/bin/activate
uv pip install pip wheel "setuptools<81"
uv pip install torch==2.13.0 torchaudio torchvision --index-url https://download.pytorch.org/whl/xpu
pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}
uv pip install -r .ci/docker/requirements-ci.txt
uv pip install -U typing_extensions
uv pip install pytest pytest-timeout pytest-xdist pytest-rerunfailures
# Stock PyTorch examples
python test/inductor/test_aot_inductor.py -k 'test_conv3d_xpu'
python test/nn/test_convolution.py -k 'test_conv3d_64bit_indexing_xpu'
# torch-xpu-ops examples
cd third_party/
git clone https://github.com/intel/torch-xpu-ops
xpu_ops_commit="$(cat xpu.txt)"
cd torch-xpu-ops
git checkout $xpu_ops_commit
cd test/xpu
python -m pytest test_nn_xpu.py -k 'test_warp_softmax_64bit_indexing_xpu_float32 or test_softmax_forward_64bit_indexing_xpu'
python -m pytest nn/test_convolution_xpu.py -k 'test_conv3d_64bit_indexing_xpu or test_conv_large_batch_1_xpu'
python -m pytest test_torch_xpu.py -k 'test_masked_scatter_large_tensor_xpu or test_cumsum_64bit_indexing_xpu'
```
Contributor guide
Research direction
Start with the listed stock PyTorch tests in test/inductor/test_aot_inductor.py and test/nn/test_convolution.py, then reproduce representative torch-xpu-ops cases such as test/xpu/test_nn_xpu.py and nn/test_convolution_xpu.py using the provided container setup. Compare B60 and PVC memory capacity and test behavior, then identify the cause of the new skips or failures and verify that affected rows no longer regress on B60 without hiding genuine memory failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100