QMCPACK / QMCPACK/qmcpack

intermittent failures of converter_test_* on crusher

Open
#4,233 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
403
Forks
154
Avg merge
1d 13h
Merged PRs (30d)
90

Description

**Describe the bug**
A clear and concise description of what the bug is.

Can converter_test_* be run in parallel on crusher? With `ctest -R converter_test_ --output-on-failure` everything passes but with `ctest -R converter_test_ -j 64 --output-on-failure` I get a bunch of test failures.

For example:
```
19/20 Test #65: converter_test_aldet1 ............***Failed Error regular expression found in output. Regex=[ FAIL] 15.99 sec
Stderr not empty
srun: Job 180229 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 180229 step creation still disabled, retrying (Requested nodes are busy)
srun: Step created for job 180229
```

**To Reproduce**
Steps to reproduce the behavior:
1. release version or git commit hash being built
commit 863c1397eea6f257c8592df36e57915de7d7c695
3. cmake command
cmake -DMPIEXEC_EXECUTABLE=`which srun` -DBOOST_ROOT=~/boost_1_78_0/ -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DENABLE_OFFLOAD=ON -DENABLE_CUDA=ON -DQMC_CUDA2HIP=ON -DHIP_ARCH=gfx90a ../
5. full program/test invocation command
ctest -R converter_test_ -j 64 --output-on-failure
7. additional steps

**Expected behavior**
A clear and concise description of what you expected to happen.

All deterministic tests pass on crusher

**System:**
crusher
- modules loaded [e.g. output of `module list`]
https://github.com/QMCPACK/qmcpack/blob/863c1397eea6f257c8592df36e57915de7d7c695/config/build_olcf_crusher_afar.sh
- other systems where this is reproducible [e.g. "my laptop", "none"]

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue on Crusher with the documented CMake configuration and compare `ctest -R converter_test_ --output-on-failure` with the `-j 64` invocation. Inspect the converter_test_* definitions and the Slurm stderr for parallel-only failures. Done means all deterministic converter tests pass when run in parallel.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
build-system, hpc, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.