[BMG] [UT] test_for_overlapping_kernels.py::TestOverlappingKernels::test_overlapping_kernels AssertionError: Scalars are not equal!
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 107
Description
### 🐛 Describe the bug
During nightly test, this profile UT case failed with below log:
```
============================= test session starts ==============================
platform linux -- Python 3.10.21, pytest-7.3.2, pluggy-1.6.0 -- /__w/torch-xpu-ops/torch-xpu-ops/.venv/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/.hypothesis/examples')
rootdir: /__w/torch-xpu-ops/torch-xpu-ops/pytorch
configfile: pytest.ini
plugins: xdist-3.3.1, subtests-0.13.1, rerunfailures-14.0, cpp-2.3.0, hypothesis-6.56.4, timeout-2.4.0, xdoctest-1.3.0, flakefinder-1.1.0
timeout: 3600.0s
timeout method: thread
timeout func_only: False
collecting ... collected 1 item
test/profiling/test_for_overlapping_kernels.py::TestOverlappingKernels::test_overlapping_kernels Found 951 overlapping kernel pairs!
Top 10 longest overlaps:
Overlap #1:
Kernel 1: ProfilerStep#19 (gpu_user_annotation) - [5877365641832.521 -> 5877367057470.267]
Kernel 2: at::native::xpu::MaxPool2dBackwardDeterministicKernelFunctor (kernel) - [5877367005647.266 -> 5877367019314.869]
Overlap duration: 13667.603515625 units
Overlap #2:
Kernel 1: ProfilerStep#19 (gpu_user_annotation) - [5877365641832.521 -> 5877367057470.267]
Kernel 2: at::native::xpu::BatchNormBackwardKernelFunctor<16, float, float, float, int> (kernel) - [5877366937863.364 -> 5877366950277.53]
Overlap duration: 12414.166015625 units
Overlap #3:
Kernel 1: ProfilerStep#19 (gpu_user_annotation) - [5877365641832.521 -> 5877367057470.267]
Kernel 2: at::native::xpu::BatchNormBackwardKernelFunctor<16, float, float, float, int> (kernel) - [5877366817615.314 -> 5877366830029.168]
Overlap duration: 12413.853515625 units
Overlap #4:
Kernel 1: ProfilerStep#19 (gpu_user_annotation) - [5877365641832.521 -> 5877367057470.267]
Kernel 2: at::native::xpu::BatchNormBackwardKernelFunctor<16, float, float, float, int> (kernel) - [5877366877709.813 -> 5877366890121.375]
Overlap duration: 12411.5615234375 units
Overlap #5:
Kernel 1: ProfilerStep#19 (gpu_user_annotation) - [5877365641832.521 -> 5877367057470.267]
Kernel 2: at::native::xpu::BatchNormBackwardKernelFunctor<16, float, float, float, int> (kernel) - [5877366955843.593 -> 5877366968236.301]
Overlap duration: 12392.7080078125 units
Overlap #6:
Kernel 1: ProfilerStep#19 (gpu_user_annotation) - [5877365641832.521 -> 5877367057470.267]
Kernel 2: at::native::xpu::BatchNormBackwardKernelFunctor<32, float, float, float, int> (kernel) - [5877367025507.976 -> 5877367035927.87]
Overlap duration: 10419.89453125 units
Overlap #7:
Kernel 1: ProfilerStep#19 (gpu_user_annotation) - [5877365641832.521 -> 5877367057470.267]
Kernel 2: conv_reorder (kernel) - [5877366729386.336 -> 5877366738600.294]
Overlap duration: 9213.9580078125 units
Overlap #8:
Kernel 1: ProfilerStep#19 (gpu_user_annotation) - [5877365641832.521 -> 5877367057470.267]
Kernel 2: gen_conv (kernel) - [5877365648483.279 -> 5877365656861.299]
Overlap duration: 8378.01953125 units
Overlap #9:
Kernel 1: ProfilerStep#19 (gpu_user_annotation) - [5877365641832.521 -> 5877367057470.267]
Kernel 2: gen_conv (kernel) - [5877367041063.713 -> 5877367049026.004]
Overlap duration: 7962.291015625 units
Overlap #10:
Kernel 1: ProfilerStep#19 (gpu_user_annotation) - [5877365641832.521 -> 5877367057470.267]
Kernel 2: conv_reorder (kernel) - [5877366522134.374 -> 5877366528689.165]
Overlap duration: 6554.791015625 units
... and 941 more overlaps
FAILED
=================================== FAILURES ===================================
_______________ TestOverlappingKernels.test_overlapping_kernels ________________
Traceback (most recent call last):
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/profiling/test_for_overlapping_kernels.py", line 130, in test_overlapping_kernels
self.assertEqual(
File "/__w/torch-xpu-ops/torch-xpu-ops/.venv/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4937, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Scalars are not equal!
Expected 0 but got 951.
Absolute difference: 951
Relative difference: inf
Found 951 overlapping kernel pairs!
To execute this test, run the following from the base repo dir:
python test/profiling/test_for_overlapping_kernels.py TestOverlappingKernels.test_overlapping_kernels
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
- generated xml file: /localdisk/home/jenkins/actions-runner/_work/torch-xpu-ops/torch-xpu-ops/ut_log/xpu_profiling/issue_reproduce/xpu_profiling_test_for_overlapping_kernels.xml -
=========================== short test summary info ============================
FAILED test/profiling/test_for_overlapping_kernels.py::TestOverlappingKernels::test_overlapping_kernels - AssertionError: Scalars are not equal!
Expected 0 but got 951.
Absolute difference: 951
Relative difference: inf
Found 951 overlapping kernel pairs!
To execute this test, run the following from the base repo dir:
python test/profiling/test_for_overlapping_kernels.py TestOverlappingKernels.test_overlapping_kernels
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
============================== 1 failed in 3.41s ===============================
```
### Versions
Pytorch: main branch
torch-xpu-ops: main branch
Contributor guide
Assessment
This issue has not been assessed yet.