E4S-Project / E4S-Project/testsuite

Veloc test fails on perlmutter

Open
#56 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Fortran
Stars
21
Forks
33
Avg merge
3m
Merged PRs (30d)
1

Description

@gonsie
@vsoch

The veloc standalone test defined here: https://github.com/E4S-Project/testsuite/tree/master/validation_tests/veloc fails when run on the veloc installed as part of the e4s 22.11 deployment on perlmutter using these variants:

-- linux-sles15-zen3 / gcc@11.2.0 -------------------------------
44htwoe veloc@1.5~ipo build_system=cmake build_type=RelWithDebInfo

With this console output:

REDSET 0.1.0 ABORT: rank 0 on nid001901: XOR requires at least 2 ranks per set, but found 1 rank(s) in set @ /tmp/lpeyrala/spack-stage/spack-stage-redset-0.2.0-4rss7cokqukwuvzvzlymxazgem3a6gim/spack-src/src/redset_xor.c:157
MPICH ERROR [Rank 0] [job id 3921177.7] [Tue Dec  6 14:52:36 2022] [nid001901] - Abort(-1) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0
srun: error: nid001901: task 0: Exited with exit code 255
srun: launch/slurm: _step_signal: Terminating StepId=3921177.7
slurmstepd: error: *** STEP 3921177.7 ON nid001901 CANCELLED AT 2022-12-06T22:52:36 ***
srun: error: nid001901: task 1: Terminated
srun: Force Terminated StepId=3921177.7

Updating to the latest heatdis_mem.c included with Veloc 1.5 resulted in this runtime error output:

[FATAL 0] [/tmp/lpeyrala/spack-stage/spack-stage-veloc-1.5-44htwoezm4qmelqjkr52pc3r3e4bqm4j/spack-src/src/lib/client.cpp:57:client_impl_t] MPI threaded mode requested but not available, please use MPI_Init_thread
[FATAL 0] [/tmp/lpeyrala/spack-stage/spack-stage-veloc-1.5-44htwoezm4qmelqjkr52pc3r3e4bqm4j/spack-src/src/lib/client.cpp:57:client_impl_t] MPI threaded mode requested but not available, please use MPI_Init_thread
[FATAL 0] [/tmp/lpeyrala/spack-stage/spack-stage-veloc-1.5-44htwoezm4qmelqjkr52pc3r3e4bqm4j/spack-src/src/lib/client.cpp:57:client_impl_t] MPI threaded mode requested but not available, please use MPI_Init_thread
[FATAL 0] [/tmp/lpeyrala/spack-stage/spack-stage-veloc-1.5-44htwoezm4qmelqjkr52pc3r3e4bqm4j/spack-src/src/lib/client.cpp:57:client_impl_t] MPI threaded mode requested but not available, please use MPI_Init_thread

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the standalone test in validation_tests/veloc and the heatdis_mem.c and client.cpp locations mentioned in the report. Reproduce the failure on perlmutter with the shown Veloc 1.5 variant, then identify the expected MPI initialization and rank behavior; done means the validation test completes without either reported runtime failure.

Written by the indexing model from the issue text.

Assessment

Domain
hpc, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.