Assert tests causing GPU resets/lockups on BMG Linux
- Dominant language
- LLVM
- Stars
- 1.5k
- Forks
- 854
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 137
Description
### Describe the bug
Just running the all tests in `run-only` mode causes GPU resets caused by the assert tests, and using the below script locks up the whole system on Battlemage.
### To reproduce
Run `assert_in_one_kernel.cpp` with the below script as follows:
```
./flake.sh /tmp/test/build/Assert/Output/assert_in_one_kernel.cpp.tmp.out 500
```
flake.sh:
```
if [[ $# < 1 ]]
then
echo "Require at least 1 argument:"
echo "1.command [2.thread number]"
echo "Example: sh xx.sh test.out 10"
echo "Example: sh test.sh \"env SYCL_DEVICE_FILTER=level_zero:gpu,host ZE_DEBUG=-1 /export/users/yuxianch/tc_test/build/SYCL/Reduction/Output/reduction_nd_ext_half.cpp.tmp.out\" 10"
else
mkdir -p test_flaky && cd test_flaky
if [[ -n $2 ]]
then
num=$2
else
num="30"
fi
cmd=$1
echo "CMD:"
echo " $cmd"
for ((i=0;i<$num;i++))
do
($cmd > out_$i 2>&1) &
pid[$i]=$!
done
for ((i=0;i<$num;i++))
do
job=${pid[$i]}
printf "${job}: "
wait ${job}; exit_code=$?
if [[ ${exit_code} != 0 ]]
then
echo "The test is failed when i = $i, exit code = ${exit_code}! Check test_flaky/out_$i"
else
printf ". "
fi
done
printf "\n\n"
fi
```
### Environment
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.