ESMCI / ESMCI/cime

Second job in a multi-submit test incorrect

Open
#3,104 18 comments 0 reactions 1 assignee Assigned to @jasonb5 View on GitHub
Assigned Low Priority ty: Bug
Dominant language
Python
Stars
174
Forks
225
Avg merge
3d 10h
Merged PRs (30d)
18

Description

For a multi-submit test like `PET.f45_g37_rx1.A`, the first job is launched on `2N` nodes because of 2 threads and the second job launched on `N` nodes with 1 thread/task. It appears that 2nd job is still using `2N` nodes.

On Anvil, a Broadwell (no hyper-threading) machine with SLURM:
`.case.test` in case1 has
```
#SBATCH --nodes=2
```
with `.case.test` in case2
```
#SBATCH --nodes=1
```

But, `stdout` shows for job1
```
0: GLOBAL communicator : 2 nodes, 36 MPI tasks
0: COMMUNICATOR NODE # [NODE NAME] : (# OF MPI TASKS) TASK # LIST
0: GLOBAL NODE 0 [ b749 ] : ( 18 MPI TASKS ) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
0: GLOBAL NODE 1 [ b750 ] : ( 18 MPI TASKS ) 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
```
and for job2
```
0: GLOBAL communicator : 2 nodes, 36 MPI tasks
0: COMMUNICATOR NODE # [NODE NAME] : (# OF MPI TASKS) TASK # LIST
0: GLOBAL NODE 0 [ b703 ] : ( 35 MPI TASKS ) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
0: GLOBAL NODE 1 [ b704 ] : ( 1 MPI TASKS ) 35
```

This can also be reproduced on Cori-KNL with latest cime master.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.