intel / intel/llvm

Assert tests causing GPU resets/lockups on BMG Linux

Open
#17,203 0 comments 0 reactions 0 assignees View on GitHub
bug confirmed
Dominant language
LLVM
Stars
1.5k
Forks
854
Avg merge
3d 17h
Merged PRs (30d)
137

Description

### Describe the bug

Just running the all tests in `run-only` mode causes GPU resets caused by the assert tests, and using the below script locks up the whole system on Battlemage.

### To reproduce

Run `assert_in_one_kernel.cpp` with the below script as follows:

```
./flake.sh /tmp/test/build/Assert/Output/assert_in_one_kernel.cpp.tmp.out 500
```

flake.sh:
```
if [[ $# < 1 ]]
then
echo "Require at least 1 argument:"
echo "1.command [2.thread number]"
echo "Example: sh xx.sh test.out 10"
echo "Example: sh test.sh \"env SYCL_DEVICE_FILTER=level_zero:gpu,host ZE_DEBUG=-1 /export/users/yuxianch/tc_test/build/SYCL/Reduction/Output/reduction_nd_ext_half.cpp.tmp.out\" 10"
else
mkdir -p test_flaky && cd test_flaky
if [[ -n $2 ]]
then
num=$2
else
num="30"
fi
cmd=$1
echo "CMD:"
echo " $cmd"

for ((i=0;i<$num;i++))
do
($cmd > out_$i 2>&1) &
pid[$i]=$!
done

for ((i=0;i<$num;i++))
do
job=${pid[$i]}
printf "${job}: "
wait ${job}; exit_code=$?
if [[ ${exit_code} != 0 ]]
then
echo "The test is failed when i = $i, exit code = ${exit_code}! Check test_flaky/out_$i"
else
printf ". "
fi
done
printf "\n\n"
fi

```

### Environment

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.