MetOffice / MetOffice/lfric_core
final_io prone to hanging
- Dominant language
- Fortran
- Stars
- 26
- Forks
- 73
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 4
Description
### Version
main
### Are there any linked Issues or Pull Requests?
_No response_
### What happened?
I've been running into issue with hanging IO on `azngarch` site. I added traces around each of the finalise calls to work out the exact point hanging and it is the call to `final_io`
```
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!> @brief Finalises infrastructure and constants used by the model.
!>
subroutine finalise_infrastructure(modeldb)
implicit none
class(modeldb_type), intent(inout) :: modeldb
! ── per-rank infrastructure-finalise trace, added to help diagnose the ─
! ── month-length chunking-test shutdown hangs - breaks down the three ──
! ── sub-stages inside finalise_infrastructure specifically. ────────────
integer :: dbg_unit, dbg_rank
character(len=64) :: dbg_path
dbg_rank = modeldb%mpi%get_comm_rank()
write(dbg_path,'("finalise_infra.",I0.5,".trace")') dbg_rank
open(newunit=dbg_unit, file=trim(dbg_path), status='replace', &
action='write', form='formatted')
call mark(dbg_unit, 'A start finalise_infrastructure')
!-------------------------------------------------------------------------
! Finalise IO
!-------------------------------------------------------------------------
call final_io( modeldb )
call mark(dbg_unit, 'B after final_io (incl. XIOS context finalise)')
```
For instance each finalise_infra trace will look like the following:
```
$ cat finalise_infra.00015.trace
A start finalise_infrastructure | VmRSS=2513744 kB
```
So never getting to point `B`
This seems to be tied to model_db size and maybe related to threading.
I consistently get it with the following model configuration:
- C12, 1 month duration, 3 ranks, 4 threads
- C48, 1 month duration, 54 ranks, 4 threads
- C48, 1 month duration, 108 ranks, 2 threads
And consistently **dont** get it with the following configuration:
- C12, 1 month duration, 6 ranks, 2 threads
- C12, 1 month duration, 12 ranks, 1 thread
- C48, 1 month duration, 216 ranks, 1 thread
XIOS not using server mode.
All using the nvfortran compiler and on the `azngarch` site - issue not seen with CCE or GNU on the `meto` site.
Regardless of issue being specific to the site there probably should be guardrails in place for a subroutine that can hang - as this can be $$ depending on the wallclock limit set.
Please ask for more information if needed.
### Relevant log output
```shell
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with finalise_infrastructure and the final_io call shown in the issue, then reproduce the hang using the listed rank/thread configurations on azngarch. Compare behavior across nvfortran, CCE, and GNU environments, using the per-rank finalise_infra traces to locate where shutdown stops. Done means the hanging condition is understood and resolved or an appropriate guardrail is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100