intel / intel/torch-xpu-ops

[Driver 26.19] New UT failures in XPU topk-related tests

Open
#3,791 4 comments 0 reactions 1 assignee Claimed by @BBBela View on GitHub
dependency component: driver hw: PVC skipped test: ut
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

## Summary
Driver 26.19 introduces 15 remaining UT failures compared with driver 25.40.

## Failing cases
- third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU::test_forward_mode_AD_topk_xpu_float64
Reproducer: `pytest -v test_ops_fwd_gradients_xpu.py -k test_forward_mode_AD_topk_xpu_float64`
- third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU::test_fn_fwgrad_bwgrad_topk_xpu_float64
Reproducer: `pytest -v test_ops_fwd_gradients_xpu.py -k test_fn_fwgrad_bwgrad_topk_xpu_float64`
- third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU::test_fn_grad_topk_xpu_float64
Reproducer: `pytest -v test_ops_gradients_xpu.py -k test_fn_grad_topk_xpu_float64`
- third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU::test_fn_gradgrad_topk_xpu_float64
Reproducer: `pytest -v test_ops_gradients_xpu.py -k test_fn_gradgrad_topk_xpu_float64`
- third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU::test_topk_integral_xpu_int64
Reproducer: `pytest -v test_sort_and_select_xpu.py -k test_topk_integral_xpu_int64`
- third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU::test_topk_nonfinite_xpu_float64
Reproducer: `pytest -v test_sort_and_select_xpu.py -k test_topk_nonfinite_xpu_float64`
- third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU::test_topk_xpu
Reproducer: `pytest -v test_sort_and_select_xpu.py -k test_topk_xpu`
- third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU::test_topk_integral_xpu_int16
Reproducer: `pytest -v test_sort_and_select_xpu.py -k test_topk_integral_xpu_int16`
- third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU::test_topk_integral_xpu_int32
Reproducer: `pytest -v test_sort_and_select_xpu.py -k test_topk_integral_xpu_int32`
- third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU::test_topk_nonfinite_xpu_float16
Reproducer: `pytest -v test_sort_and_select_xpu.py -k test_topk_nonfinite_xpu_float16`
- third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU::test_topk_lower_precision_xpu_bfloat16
Reproducer: `pytest -v test_sort_and_select_xpu.py -k test_topk_lower_precision_xpu_bfloat16`
- third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU::test_topk_nonfinite_xpu_float32
Reproducer: `pytest -v test_sort_and_select_xpu.py -k test_topk_nonfinite_xpu_float32`
- third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU::test_topk_nonfinite_xpu_bfloat16
Reproducer: `pytest -v test_sort_and_select_xpu.py -k test_topk_nonfinite_xpu_bfloat16`
- third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU::test_topk_lower_precision_xpu_float16
Reproducer: `pytest -v test_sort_and_select_xpu.py -k test_topk_lower_precision_xpu_float16`
- third_party.torch-xpu-ops.test.xpu.test_ops_xpu.TestCommonXPU::test_compare_cpu_topk_xpu_float32
Reproducer: `pytest -v test_ops_xpu.py -k test_compare_cpu_topk_xpu_float32`

## Error excerpts

### test_forward_mode_AD_topk_xpu_float64
```text
Exception: Jacobian computed with forward mode mismatch for output 0 with respect to input 0,
numerical:tensor([ 3.0415e-02, 4.5859e-02, 3.8152e-02, 1.1054e-01, 8.1856e-03,
8.1320e-02, 4
```

### test_fn_fwgrad_bwgrad_topk_xpu_float64
```text
Exception: Jacobian computed with forward mode mismatch for output 0 with respect to input 0,
numerical:tensor([ 0.0000, 0.0000, 0.0000, -481235.1729, 481235.1729,
0.00
```

### test_fn_grad_topk_xpu_float64
```text
Exception: Jacobian mismatch for output 0 with respect to input 0,
numerical:290875.32303185354
analytical:0.5674704909096284

The above quantities relating the numerical and analytical jacobians are
```

### test_fn_gradgrad_topk_xpu_float64
```text
Exception: Jacobian mismatch for output 0 with respect to input 0,
numerical:-8806.595901846638
analytical:0.0

The above quantities relating the numerical and analytical jacobians are computed
in fa
```

### test_topk_integral_xpu_int64
```text
AssertionError: Tensor-likes are not equal!

Mismatched elements: 5 / 5 (100.0%)
Greatest absolute difference: 7709152990306715893 at index (4,)
Greatest relative difference: 4.254495620727539 at inde
```

### test_topk_nonfinite_xpu_float64
```text
AssertionError: Tensor-likes are not close!

Mismatched elements: 3 / 4 (75.0%)
Greatest absolute difference: inf at index (1,) (up to 1e-07 allowed)
Greatest relative difference: nan at index (1,) (u
```

### test_topk_xpu
```text
AssertionError: Tensor-likes are not equal!

Mismatched elements: 8850 / 62700 (14.1%)
Greatest absolute difference: 0.35633307695388794 at index (10, 37, 18)
Greatest relative difference: 2.633792161
```

### test_topk_integral_xpu_int16
```text
AssertionError: Tensor-likes are not equal!

Mismatched elements: 4 / 5 (80.0%)
Greatest absolute difference: 19656 at index (2,)
Greatest relative difference: 2.124054431915283 at index (2,)

To exec
```

### test_topk_integral_xpu_int32
```text
AssertionError: Tensor-likes are not equal!

Mismatched elements: 2 / 5 (40.0%)
Greatest absolute difference: 1794927053 at index (4,)
Greatest relative difference: 1.702272653579712 at index (3,)

To
```

### test_topk_nonfinite_xpu_float16
```text
AssertionError: Tensor-likes are not close!

Mismatched elements: 3 / 4 (75.0%)
Greatest absolute difference: inf at index (1,) (up to 1e-05 allowed)
Greatest relative difference: nan at index (1,) (u
```

### test_topk_lower_precision_xpu_bfloat16
```text
AssertionError: Tensor-likes are not close!

Mismatched elements: 4 / 5 (80.0%)
Greatest absolute difference: 1.078125 at index (1,) (up to 1e-05 allowed)
Greatest relative difference: 2.109375 at ind
```

### test_topk_nonfinite_xpu_float32
```text
AssertionError: Tensor-likes are not close!

Mismatched elements: 3 / 4 (75.0%)
Greatest absolute difference: inf at index (1,) (up to 1e-05 allowed)
Greatest relative difference: nan at index (1,) (u
```

### test_topk_nonfinite_xpu_bfloat16
```text
AssertionError: Tensor-likes are not close!

Mismatched elements: 3 / 4 (75.0%)
Greatest absolute difference: inf at index (1,) (up to 1e-05 allowed)
Greatest relative difference: nan at index (1,) (u
```

### test_topk_lower_precision_xpu_float16
```text
AssertionError: Tensor-likes are not close!

Mismatched elements: 5 / 5 (100.0%)
Greatest absolute difference: 1.0830078125 at index (4,) (up to 1e-05 allowed)
Greatest relative difference: 2.11328125
```

### test_compare_cpu_topk_xpu_float32
```text
Exception: Tensor-likes are not close!

Mismatched elements: 104 / 150 (69.3%)
Greatest absolute difference: 12.189497947692871 at index (4, 6, 2) (up to 0.001 allowed)
Greatest relative difference: 1
```

## Reproducer
### Environment setup
```bash
docker run -it -e TZ=Aisa/Shanghai --device=/dev/mem --device=/dev/dri --group-add video --privileged --shm-size=8g intelgpu/ubuntu-24.04-lts2:2523.40 bash

curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
uv venv myvenv --python 3.12 --clear
source myvenv/bin/activate
uv pip install pip setuptools wheel
uv pip install torch torchaudio torchvision --index-url https://download.pytorch.org/whl/xpu

pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}

uv pip install pip pandas psutil scipy requests
uv pip install -U numpy==1.26.4 accelerate
uv pip install -r .ci/docker/ci_commit_pins/huggingface-requirements.txt
uv pip install timm

uv pip install -r .ci/docker/requirements-ci.txt
uv pip install -U typing_extensions

# update torch
# gh --repo intel/torch-xpu-ops run download 26399638675 -p "Torch-XPU-Wheel-*"
uv pip uninstall torch torchaudio torchvision triton-xpu -y
uv pip install Torch-XPU-Wheel-*/*.whl
```

### Run individual failing tests
Use any of the pytest reproducer commands listed above.

## Source runs
- Target: https://github.com/intel/torch-xpu-ops/actions/runs/26264076598
- Baseline: https://github.com/intel/torch-xpu-ops/actions/runs/26267132121

Cases:
~~op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_forward_mode_AD_topk_xpu_float64~~
~~op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_fn_fwgrad_bwgrad_topk_xpu_float64~~
~~op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_grad_topk_xpu_float64~~
~~op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_gradgrad_topk_xpu_float64~~
op_ut,third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU,test_topk_nonfinite_xpu_float16
op_ut,third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU,test_topk_integral_xpu_int16
op_ut,third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU,test_topk_integral_xpu_int32
op_ut,third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU,test_topk_nonfinite_xpu_float32
op_ut,third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU,test_topk_nonfinite_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU,test_topk_integral_xpu_int64
op_ut,third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU,test_topk_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU,test_topk_lower_precision_xpu_float16
op_ut,third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU,test_topk_lower_precision_xpu_bfloat16
op_ut,third_party.torch-xpu-ops.test.xpu.test_sort_and_select_xpu.TestSortAndSelectXPU,test_topk_nonfinite_xpu_bfloat16
~~op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_xpu.TestCommonXPU,test_compare_cpu_topk_xpu_float32~~

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.