MetOffice / MetOffice/lfric_apps
Sporadic failures in mpi_bcast
- Dominant language
- Fortran
- Stars
- 31
- Forks
- 118
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 8
Description
### Version
main
### Are there any linked Issues or Pull Requests?
_No response_
### What happened?
GC6 with LFRic at the git_migration release fails sporadically with "rank 510 died from signal 6 and dumped core". I have interrogated the core file with ddt and the traceback shows that it is the call to [mpi_bcast](https://code.metoffice.gov.uk/trac/nemo/browser/NEMO/branches/dev/dancopsey/NEMO_4.0.4_GOSI9_new_sea_meltpond_coupling/src/OCE/SBC/cpl_oasis3.F90#L704) in NEMO's file [cpl_oasis3.F90](https://code.metoffice.gov.uk/trac/nemo/browser/NEMO/branches/dev/dancopsey/NEMO_4.0.4_GOSI9_new_sea_meltpond_coupling/src/OCE/SBC/cpl_oasis3.F90) (subroutine cpl_rcv_1d). This is when NEMO receives 1D data from OASIS (ice mass or river outflow) on processor zero and then passes it to all the other NEMO processors using mpi_bcast. This has only started happening after we have [upgraded the modules in the GC6 trunk workflow](https://code.metoffice.gov.uk/trac/roses-u/changeset/346748/d/i/6/2/0/trunk/site/meto-excd.cylc) which I think involves an upgrade to MPICH. I have tried replacing mpi_bcast with mpi_send (from processor zero) and mpi_recv (from all the other processors) and it still fails with a similar error however it appears to fail in one of the last NEMO processors when it is in mpi_recv while it is waiting for processor zero to get to the mpi_send stage.
### Relevant log output
```shell
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at NEMO's cpl_oasis3.F90, specifically subroutine cpl_rcv_1d and the mpi_bcast call around line 704. Reproduce the sporadic GC6 LFRic failure at the git_migration release, then compare the MPI module upgrade and the reported mpi_send/mpi_recv behavior. Done means identifying and correcting the communication failure, with the workflow completing reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- distributed-systems, hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100