Cache tests fail after Linux GPU reset
- Dominant language
- LLVM
- Stars
- 1.5k
- Forks
- 854
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 137
Description
### Describe the bug
If we start doing a GPU reset before starting E2E tests on Linux BMG, these tests [fail](https://github.com/intel/llvm/actions/runs/13816578232/job/38668561737?pr=17381) for some reason
```
Failed Tests (3):
SYCL :: KernelAndProgram/persistent-cache-multi-device.cpp
SYCL :: ProgramManager/multi_device_bundle/build_twice.cpp
SYCL :: ProgramManager/multi_device_bundle/device_libs_and_caching.cpp
```
```
FAIL: SYCL :: ProgramManager/multi_device_bundle/device_libs_and_caching.cpp (530 of 2285)
******************** TEST 'SYCL :: ProgramManager/multi_device_bundle/device_libs_and_caching.cpp' FAILED ********************
Exit Code: 1
Command Output (stderr):
--
RUN: at line 17: env NEOReadDebugKeys=1 CreateMultipleRootDevices=4 SYCL_UR_TRACE=2 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/ProgramManager/multi_device_bundle/Output/device_libs_and_caching.cpp.tmp_jit.out | /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/device_libs_and_caching.cpp --check-prefixes=CHECK-SPIRV-JIT-LINK-TRACE
+ env NEOReadDebugKeys=1 CreateMultipleRootDevices=4 SYCL_UR_TRACE=2 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/ProgramManager/multi_device_bundle/Output/device_libs_and_caching.cpp.tmp_jit.out
+ /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/device_libs_and_caching.cpp --check-prefixes=CHECK-SPIRV-JIT-LINK-TRACE
UR ---> platform->initialize()
UR <--- platform->initialize()(UR_RESULT_SUCCESS)
UR ---> Device->initialize()
UR <--- Device->initialize()(UR_RESULT_SUCCESS)
/__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/device_libs_and_caching.cpp:68:33: error: CHECK-SPIRV-JIT-LINK-TRACE: expected string not found in input
// CHECK-SPIRV-JIT-LINK-TRACE: urProgramCreateWithIL(
^
:1:1: note: scanning from here
---> urAdapterGet
^
:1:7: note: possible intended match here
---> urAdapterGet
^
Input file:
Check file: /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/device_libs_and_caching.cpp
-dump-input=help explains the following input dump.
Input was:
<<<<<<
1: ---> urAdapterGet
check:68'0 X~~~~~~~~~~~~~~~~~~ error: no match found
check:68'1 ? possible intended match
2: <--- urAdapterGet(.NumEntries = 0, .phAdapters = nullptr, .pNumAdapters = 0x7ffe6c5ee704 (1)) -> UR_RESULT_SUCCESS;
check:68'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3: ---> urAdapterGet
check:68'0 ~~~~~~~~~~~~~~~~~~~
4: <--- urAdapterGet(.NumEntries = 1, .phAdapters = 0x8037f30 {0x8036670}, .pNumAdapters = nullptr) -> UR_RESULT_SUCCESS;
check:68'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5: ---> urAdapterGetInfo
check:68'0 ~~~~~~~~~~~~~~~~~~~~~~~
6: <--- urAdapterGetInfo(.hAdapter = 0x8036670, .propName = UR_ADAPTER_INFO_BACKEND, .propSize = 4, .pPropValue = 0x7ffe6c5ee6f8 (UR_ADAPTER_BACKEND_LEVEL_ZERO), .pPropSizeRet = nullptr) -> UR_RESULT_SUCCESS;
check:68'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.
.
.
>>>>>>
--
********************
FAIL: SYCL :: KernelAndProgram/persistent-cache-multi-device.cpp (1246 of 2285)
******************** TEST 'SYCL :: KernelAndProgram/persistent-cache-multi-device.cpp' FAILED ********************
Exit Code: 1
Command Output (stderr):
--
RUN: at line 5: env NEOReadDebugKeys=1 CreateMultipleRootDevices=3 SYCL_CACHE_PERSISTENT=1 SYCL_CACHE_TRACE=1 SYCL_CACHE_DIR=/__w/llvm/llvm/build-e2e/KernelAndProgram/Output/persistent-cache-multi-device.cpp.tmp/cache_dir env -u XDG_CACHE_HOME env -u HOME env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/KernelAndProgram/Output/persistent-cache-multi-device.cpp.tmp.out 2>&1 | /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp --check-prefixes=CHECK-BUILD
+ env NEOReadDebugKeys=1 CreateMultipleRootDevices=3 SYCL_CACHE_PERSISTENT=1 SYCL_CACHE_TRACE=1 SYCL_CACHE_DIR=/__w/llvm/llvm/build-e2e/KernelAndProgram/Output/persistent-cache-multi-device.cpp.tmp/cache_dir env -u XDG_CACHE_HOME env -u HOME env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/KernelAndProgram/Output/persistent-cache-multi-device.cpp.tmp.out
+ /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp --check-prefixes=CHECK-BUILD
/__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp:30:18: error: CHECK-BUILD: expected string not found in input
// CHECK-BUILD: [Persistent Cache]: device binary has been cached
^
:1:1: note: scanning from here
persistent-cache-multi-device.cpp.tmp.out: /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp:25: int main(): Assertion `devs.size() >= 3' failed.
^
:1:94: note: possible intended match here
persistent-cache-multi-device.cpp.tmp.out: /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp:25: int main(): Assertion `devs.size() >= 3' failed.
^
Input file:
Check file: /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp
-dump-input=help explains the following input dump.
Input was:
<<<<<<
1: persistent-cache-multi-device.cpp.tmp.out: /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp:25: int main(): Assertion `devs.size() >= 3' failed.
check:30'0 X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:30'1 ? possible intended match
>>>>>>
--
********************
FAIL: SYCL :: ProgramManager/multi_device_bundle/build_twice.cpp (1397 of 2285)
******************** TEST 'SYCL :: ProgramManager/multi_device_bundle/build_twice.cpp' FAILED ********************
Exit Code: 1
Command Output (stderr):
--
RUN: at line 7: env NEOReadDebugKeys=1 CreateMultipleRootDevices=3 SYCL_UR_TRACE=2 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/ProgramManager/multi_device_bundle/Output/build_twice.cpp.tmp.out | /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/build_twice.cpp
+ env NEOReadDebugKeys=1 CreateMultipleRootDevices=3 SYCL_UR_TRACE=2 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/ProgramManager/multi_device_bundle/Output/build_twice.cpp.tmp.out
+ /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/build_twice.cpp
UR ---> platform->initialize()
UR <--- platform->initialize()(UR_RESULT_SUCCESS)
UR ---> Device->initialize()
UR <--- Device->initialize()(UR_RESULT_SUCCESS)
/__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/build_twice.cpp:32:12: error: CHECK: expected string not found in input
// CHECK: urProgramCreateWithIL(
^
:1:1: note: scanning from here
---> urAdapterGet
^
:1:7: note: possible intended match here
---> urAdapterGet
^
Input file:
Check file: /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/build_twice.cpp
-dump-input=help explains the following input dump.
Input was:
<<<<<<
1: ---> urAdapterGet
check:32'0 X~~~~~~~~~~~~~~~~~~ error: no match found
check:32'1 ? possible intended match
2: <--- urAdapterGet(.NumEntries = 0, .phAdapters = nullptr, .pNumAdapters = 0x7ffe0a7e85b4 (1)) -> UR_RESULT_SUCCESS;
check:32'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3: ---> urAdapterGet
check:32'0 ~~~~~~~~~~~~~~~~~~~
4: <--- urAdapterGet(.NumEntries = 1, .phAdapters = 0x200561a0 {0x200548e0}, .pNumAdapters = nullptr) -> UR_RESULT_SUCCESS;
check:32'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5: ---> urAdapterGetInfo
check:32'0 ~~~~~~~~~~~~~~~~~~~~~~~
6: <--- urAdapterGetInfo(.hAdapter = 0x200548e0, .propName = UR_ADAPTER_INFO_BACKEND, .propSize = 4, .pPropValue = 0x7ffe0a7e85a8 (UR_ADAPTER_BACKEND_LEVEL_ZERO), .pPropSizeRet = nullptr) -> UR_RESULT_SUCCESS;
check:32'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.
.
.
>>>>>>
--
```
### To reproduce
1. Include a code snippet that is as short as possible
2. Specify the command which should be used to compile the program
3. Specify the command which should be used to launch the program
4. Indicate what is wrong and what was expected
### Environment
- OS: [e.g Windows/Linux]
- Target device and vendor: [e.g. Intel GPU]
- DPC++ version: [e.g. commit hash or output of `clang++ --version`]
- Dependencies version: [e.g. the output of `sycl-ls --verbose`]
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.