intel / intel/torch-xpu-ops

[B60][XPU] Large tensor tests newly fail/skip with insufficient XPU memory vs PVC

Open
#4,797 0 comments 0 reactions 0 assignees View on GitHub
hw: BMG test: ut
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

## Summary

B60 reports multiple XPU memory-capacity failures/skips while the same rows passed on PVC.

## Failure Category

Insufficient XPU memory for large tensor / 64-bit indexing tests.

Representative messages:

```text
Insufficient xpu memory
Insufficient xpu:0 memory
insufficient memory
```

## Scope

- Test suites: stock PyTorch and torch-xpu-ops
- Platform comparison: B60 skipped/failed; PVC passed
- Affected rows observed: 25

## Representative Stock PyTorch Rows

```text
test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_conv3d_xpu
Skipped: insufficient memory

test/inductor/test_mix_order_reduction.py::MixOrderReductionTest::test_out_of_shared_memory
Skipped: Insufficient xpu memory

test/nn/test_convolution.py::TestConvolutionNNDeviceTypeXPU::test_conv3d_64bit_indexing_xpu
Skipped: Insufficient xpu:0 memory
```

## Representative torch-xpu-ops Rows

Direct torch-xpu-ops XPU test files:

```text
test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_warp_softmax_64bit_indexing_xpu_float32
test/xpu/nn/test_convolution_xpu.py::TestConvolutionNNDeviceTypeXPU::test_conv3d_64bit_indexing_xpu
test/xpu/nn/test_convolution_xpu.py::TestConvolutionNNDeviceTypeXPU::test_conv_large_batch_1_xpu
test/xpu/test_foreach_xpu.py::TestForeachXPU::test_foreach_copy_with_multi_dtypes_large_input_xpu
test/xpu/test_linalg_xpu.py::TestLinalgXPU::test_triu_tril_large_matrix_64bit_xpu
test/xpu/test_sparse_xpu.py::TestSparseXPU::test_coalesce_accepts_large_tensor_xpu_float32
test/xpu/test_tensor_creation_ops_xpu.py::TestRandomTensorCreationXPU::test_randperm_large_xpu
test/xpu/test_torch_xpu.py::TestTorchDeviceTypeXPU::test_masked_scatter_large_tensor_xpu
```

## Expected Behavior

These rows should not newly skip/fail for insufficient XPU memory on B60 when PVC passes.

## Notes

This may be environment/device-memory capacity rather than operator correctness, but it is tracked separately because the message family is consistent and affects many large-tensor and 64-bit indexing tests.

## Full Case List

Total cases: 25

Show all 25 cases

1. `stock pytorch` | `test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_conv3d_xpu` | `skipped`
- `insufficient memory`
2. `stock pytorch` | `test/inductor/test_mix_order_reduction.py::MixOrderReductionTest::test_out_of_shared_memory` | `skipped`
- `Insufficient xpu memory`
3. `stock pytorch` | `test/nn/test_convolution.py::TestConvolutionNNDeviceTypeXPU::test_conv3d_64bit_indexing_xpu` | `skipped`
- `Insufficient xpu:0 memory`
4. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_warp_softmax_64bit_indexing_xpu_float32` | `failed`
- `Insufficient xpu memory`
5. `torch-xpu-ops` | `test/xpu/nn/test_convolution_xpu.py::TestConvolutionNNDeviceTypeXPU::test_conv3d_64bit_indexing_xpu` | `skipped`
- `Insufficient xpu memory`
6. `torch-xpu-ops` | `test/xpu/nn/test_convolution_xpu.py::TestConvolutionNNDeviceTypeXPU::test_conv_large_batch_1_xpu` | `skipped`
- `Insufficient xpu memory`
7. `torch-xpu-ops` | `test/xpu/test_foreach_xpu.py::TestForeachXPU::test_foreach_copy_with_multi_dtypes_large_input_xpu` | `skipped`
- `Insufficient xpu memory`
8. `torch-xpu-ops` | `test/xpu/test_linalg_xpu.py::TestLinalgXPU::test_triu_tril_large_matrix_64bit_xpu` | `skipped`
- `Insufficient xpu memory`
9. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_avg_pool_large_tensor2_xpu` | `skipped`
- `Insufficient xpu memory`
10. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_large_max_pool_contig_xpu` | `skipped`
- `Insufficient xpu memory`
11. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_nll_loss_large_tensor_reduction_mean_xpu` | `skipped`
- `Insufficient xpu memory`
12. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_nll_loss_large_tensor_reduction_none_xpu` | `skipped`
- `Insufficient xpu memory`
13. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_nll_loss_large_tensor_reduction_sum_xpu` | `skipped`
- `Insufficient xpu memory`
14. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_softmax_forward_64bit_indexing_xpu` | `skipped`
- `Insufficient xpu memory`
15. `torch-xpu-ops` | `test/xpu/test_sparse_xpu.py::TestSparseXPU::test_coalesce_accepts_large_tensor_xpu_float32` | `skipped`
- `Insufficient xpu memory`
16. `torch-xpu-ops` | `test/xpu/test_tensor_creation_ops_xpu.py::TestRandomTensorCreationXPU::test_randperm_large_xpu` | `skipped`
- `Insufficient xpu memory`
17. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNN::test_layer_norm_large_tensor` | `skipped`
- `Insufficient xpu memory`
18. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_replicatepad_64bit_indexing_xpu_float16` | `skipped`
- `Insufficient xpu:0 memory`
19. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_upsampling_64bit_indexing_channels_last_xpu_bfloat16` | `skipped`
- `Insufficient xpu:0 memory`
20. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_upsampling_64bit_indexing_channels_last_xpu_float16` | `skipped`
- `Insufficient xpu:0 memory`
21. `torch-xpu-ops` | `test/xpu/test_nn_xpu.py::TestNNDeviceTypeXPU::test_upsamplingnearest2d_backward_64bit_indexing_xpu_float16` | `skipped`
- `Insufficient xpu:0 memory`
22. `torch-xpu-ops` | `test/xpu/test_tensor_creation_ops_xpu.py::TestLikeTensorCreationXPU::test_ones_large_xpu_complex128` | `skipped`
- `Insufficient xpu:0 memory`
23. `torch-xpu-ops` | `test/xpu/test_tensor_creation_ops_xpu.py::TestLikeTensorCreationXPU::test_zeros_large_xpu_complex128` | `skipped`
- `Insufficient xpu:0 memory`
24. `torch-xpu-ops` | `test/xpu/test_torch_xpu.py::TestTorchDeviceTypeXPU::test_cumsum_64bit_indexing_xpu` | `skipped`
- `Insufficient xpu:0 memory`
25. `torch-xpu-ops` | `test/xpu/test_torch_xpu.py::TestTorchDeviceTypeXPU::test_masked_scatter_large_tensor_xpu` | `skipped`
- `Insufficient xpu:0 memory`

## Reproducer / Environment Setup

```bash
docker run -it -e TZ=Asia/Shanghai --device=/dev/mem --device=/dev/dri --group-add video --privileged -v $(realpath ${HOME}):/home/jenkins --shm-size=8g intelgpu/ubuntu-26.04-rolling:26.18 bash

apt update
apt upgrade -y

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv myvenv --python 3.10 --clear
source myvenv/bin/activate
uv pip install pip wheel "setuptools<81"
uv pip install torch==2.13.0 torchaudio torchvision --index-url https://download.pytorch.org/whl/xpu

pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}

uv pip install -r .ci/docker/requirements-ci.txt
uv pip install -U typing_extensions
uv pip install pytest pytest-timeout pytest-xdist pytest-rerunfailures

# Stock PyTorch examples
python test/inductor/test_aot_inductor.py -k 'test_conv3d_xpu'
python test/nn/test_convolution.py -k 'test_conv3d_64bit_indexing_xpu'

# torch-xpu-ops examples
cd third_party/
git clone https://github.com/intel/torch-xpu-ops
xpu_ops_commit="$(cat xpu.txt)"
cd torch-xpu-ops
git checkout $xpu_ops_commit
cd test/xpu

python -m pytest test_nn_xpu.py -k 'test_warp_softmax_64bit_indexing_xpu_float32 or test_softmax_forward_64bit_indexing_xpu'
python -m pytest nn/test_convolution_xpu.py -k 'test_conv3d_64bit_indexing_xpu or test_conv_large_batch_1_xpu'
python -m pytest test_torch_xpu.py -k 'test_masked_scatter_large_tensor_xpu or test_cumsum_64bit_indexing_xpu'
```

Contributor guide

Open the contributing guide

Research direction

Start with the listed stock PyTorch tests in test/inductor/test_aot_inductor.py and test/nn/test_convolution.py, then reproduce representative torch-xpu-ops cases such as test/xpu/test_nn_xpu.py and nn/test_convolution_xpu.py using the provided container setup. Compare B60 and PVC memory capacity and test behavior, then identify the cause of the new skips or failures and verify that affected rows no longer regress on B60 without hiding genuine memory failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.