intermittent failures of converter_test_* on crusher
- Dominant language
- C++
- Stars
- 403
- Forks
- 154
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 90
Description
**Describe the bug**
A clear and concise description of what the bug is.
Can converter_test_* be run in parallel on crusher? With `ctest -R converter_test_ --output-on-failure` everything passes but with `ctest -R converter_test_ -j 64 --output-on-failure` I get a bunch of test failures.
For example:
```
19/20 Test #65: converter_test_aldet1 ............***Failed Error regular expression found in output. Regex=[ FAIL] 15.99 sec
Stderr not empty
srun: Job 180229 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Step created for job 180229
```
**To Reproduce**
Steps to reproduce the behavior:
1. release version or git commit hash being built
commit 863c1397eea6f257c8592df36e57915de7d7c695
3. cmake command
cmake -DMPIEXEC_EXECUTABLE=`which srun` -DBOOST_ROOT=~/boost_1_78_0/ -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DENABLE_OFFLOAD=ON -DENABLE_CUDA=ON -DQMC_CUDA2HIP=ON -DHIP_ARCH=gfx90a ../
5. full program/test invocation command
ctest -R converter_test_ -j 64 --output-on-failure
7. additional steps
**Expected behavior**
A clear and concise description of what you expected to happen.
All deterministic tests pass on crusher
**System:**
crusher
- modules loaded [e.g. output of `module list`]
https://github.com/QMCPACK/qmcpack/blob/863c1397eea6f257c8592df36e57915de7d7c695/config/build_olcf_crusher_afar.sh
- other systems where this is reproducible [e.g. "my laptop", "none"]
**Additional context**
Add any other context about the problem here.
Contributor guide
Research direction
Start by reproducing the issue on Crusher with the documented CMake configuration and compare `ctest -R converter_test_ --output-on-failure` with the `-j 64` invocation. Inspect the converter_test_* definitions and the Slurm stderr for parallel-only failures. Done means all deterministic converter tests pass when run in parallel.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp
- Domain
- build-system, hpc, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100