MetOffice / MetOffice/lfric_apps

Sporadic failures in mpi_bcast

Open
#269 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Fortran
Stars
31
Forks
118
Avg merge
2d 4h
Merged PRs (30d)
8

Description

### Version

main

### Are there any linked Issues or Pull Requests?

_No response_

### What happened?

GC6 with LFRic at the git_migration release fails sporadically with "rank 510 died from signal 6 and dumped core". I have interrogated the core file with ddt and the traceback shows that it is the call to [mpi_bcast](https://code.metoffice.gov.uk/trac/nemo/browser/NEMO/branches/dev/dancopsey/NEMO_4.0.4_GOSI9_new_sea_meltpond_coupling/src/OCE/SBC/cpl_oasis3.F90#L704) in NEMO's file [cpl_oasis3.F90](https://code.metoffice.gov.uk/trac/nemo/browser/NEMO/branches/dev/dancopsey/NEMO_4.0.4_GOSI9_new_sea_meltpond_coupling/src/OCE/SBC/cpl_oasis3.F90) (subroutine cpl_rcv_1d). This is when NEMO receives 1D data from OASIS (ice mass or river outflow) on processor zero and then passes it to all the other NEMO processors using mpi_bcast. This has only started happening after we have [upgraded the modules in the GC6 trunk workflow](https://code.metoffice.gov.uk/trac/roses-u/changeset/346748/d/i/6/2/0/trunk/site/meto-excd.cylc) which I think involves an upgrade to MPICH. I have tried replacing mpi_bcast with mpi_send (from processor zero) and mpi_recv (from all the other processors) and it still fails with a similar error however it appears to fail in one of the last NEMO processors when it is in mpi_recv while it is waiting for processor zero to get to the mpi_send stage.

### Relevant log output

```shell

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at NEMO's cpl_oasis3.F90, specifically subroutine cpl_rcv_1d and the mpi_bcast call around line 704. Reproduce the sporadic GC6 LFRic failure at the git_migration release, then compare the MPI module upgrade and the reported mpi_send/mpi_recv behavior. Done means identifying and correcting the communication failure, with the workflow completing reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
distributed-systems, hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.