MetOffice / MetOffice/lfric_core
Auto partitioner fails for some MPI sizes
- Dominant language
- Fortran
- Stars
- 26
- Forks
- 73
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 4
Description
### Version
main
### Are there any linked Issues or Pull Requests?
https://github.com/MetOffice/jjdocs/issues/563
### What happened?
While testing the JADA configuration for the benchmark, I encountered an issue with some of the domain decomposition sizes, namely: 336, 672 and 2352. The issue is encountered when I use the `auto partitioner` and I managed to get them to run when using a `custom partitioner`. The JADA application has a `C896 pseudo model` that reads a previously run forecast and a `C224 linear-adjoint` model that includes 3 levels of multigrid (`C112`, `C56`, `C28`).
The JADA application was run with the following resource:
`RANKS:` `Nodes/PPN/OPM_NUM_THREAD`
`336:` `21/16/8` (JADA) + `128:` `4/32/1` (XIOS).
`672:` `42/16/4` (JADA) + `128:` `4/32/1` (XIOS).
`2352:` `84/28/2` (JADA) + `128:` `4/32/1` (XIOS).
The failure appears to be a segfault. I message Tom Melvin about this and he made some suggestion about the origin of this error that is documented in: https://github.com/MetOffice/jjdocs/issues/563#issuecomment-4169103833.
Its worth noting that in the 336 case, I had originally encountered an error trap when run back in October:
`20251115235223.923+0000:P001:ERROR: Total ranks per panel 112 must be the product of xprocs 4 and yprocs 14`.
This particular error seemed to go away after the `LFRic 3.0 release`, I believe that it is related to this change: https://code.metoffice.gov.uk/trac/lfric_apps/ticket/976. I had not tested with `672` or `2352` at that time.
_Working custom partitions_
**336:**
```
&partitioning
panel_decomposition='custom'
panel_xproc=4
panel_yproc=14
```
**672:**
```
&partitioning
panel_decomposition='custom',
panel_xproc=4,
panel_yproc=28,
```
**2352:**
```
&partitioning
panel_decomposition='custom',
panel_xproc=14,
panel_yproc=28,
```
### Relevant log output
```shell
nidc1287: rank 523 died from signal 11
nidc1100: rank 131 died from signal 15
nidc1302: rank 541 died from signal 9
Running on nid 1053
Running on xname x1020c1s5b0n1
[FAIL] jada "$ROSE_DATAC" "$MODEL_TYPE" "$OBS_TASKS_ACTIVE" "$OBS_OUTPUT" # return-code=143
2026-03-30T23:33:27Z CRITICAL - failed/EXIT
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.