E4S-Project / E4S-Project/testsuite
Veloc test fails on perlmutter
Nobody has claimed this yet.
- Dominant language
- Fortran
- Stars
- 21
- Forks
- 33
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
@gonsie
@vsoch
The veloc standalone test defined here: https://github.com/E4S-Project/testsuite/tree/master/validation_tests/veloc fails when run on the veloc installed as part of the e4s 22.11 deployment on perlmutter using these variants:
-- linux-sles15-zen3 / gcc@11.2.0 -------------------------------
44htwoe veloc@1.5~ipo build_system=cmake build_type=RelWithDebInfo
With this console output:
REDSET 0.1.0 ABORT: rank 0 on nid001901: XOR requires at least 2 ranks per set, but found 1 rank(s) in set @ /tmp/lpeyrala/spack-stage/spack-stage-redset-0.2.0-4rss7cokqukwuvzvzlymxazgem3a6gim/spack-src/src/redset_xor.c:157
MPICH ERROR [Rank 0] [job id 3921177.7] [Tue Dec 6 14:52:36 2022] [nid001901] - Abort(-1) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0
srun: error: nid001901: task 0: Exited with exit code 255
srun: launch/slurm: _step_signal: Terminating StepId=3921177.7
slurmstepd: error: *** STEP 3921177.7 ON nid001901 CANCELLED AT 2022-12-06T22:52:36 ***
srun: error: nid001901: task 1: Terminated
srun: Force Terminated StepId=3921177.7
Updating to the latest heatdis_mem.c included with Veloc 1.5 resulted in this runtime error output:
[FATAL 0] [/tmp/lpeyrala/spack-stage/spack-stage-veloc-1.5-44htwoezm4qmelqjkr52pc3r3e4bqm4j/spack-src/src/lib/client.cpp:57:client_impl_t] MPI threaded mode requested but not available, please use MPI_Init_thread
[FATAL 0] [/tmp/lpeyrala/spack-stage/spack-stage-veloc-1.5-44htwoezm4qmelqjkr52pc3r3e4bqm4j/spack-src/src/lib/client.cpp:57:client_impl_t] MPI threaded mode requested but not available, please use MPI_Init_thread
[FATAL 0] [/tmp/lpeyrala/spack-stage/spack-stage-veloc-1.5-44htwoezm4qmelqjkr52pc3r3e4bqm4j/spack-src/src/lib/client.cpp:57:client_impl_t] MPI threaded mode requested but not available, please use MPI_Init_thread
[FATAL 0] [/tmp/lpeyrala/spack-stage/spack-stage-veloc-1.5-44htwoezm4qmelqjkr52pc3r3e4bqm4j/spack-src/src/lib/client.cpp:57:client_impl_t] MPI threaded mode requested but not available, please use MPI_Init_thread
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the standalone test in validation_tests/veloc and the heatdis_mem.c and client.cpp locations mentioned in the report. Reproduce the failure on perlmutter with the shown Veloc 1.5 variant, then identify the expected MPI initialization and rank behavior; done means the validation test completes without either reported runtime failure.
Written by the indexing model from the issue text.
Assessment
- Domain
- hpc, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100