MetOffice / MetOffice/lfric_apps
Input Data parallelisation
- Dominant language
- Fortran
- Stars
- 31
- Forks
- 118
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 8
Description
Analyses of the interactions with parallel file systems at scale demonstrate that `lfric_atm` is not scaling well to large rank counts.
`lfric_atm` requires significant quantities of data to initialise the model, including configurations, mesh definitions, start dumps & ancillaries.
Some facets of the code base implement simple storage interaction patterns where each rank opens a file as it needs it and obtains the information it requires. This works fine for small rank counts and okay for medium rank counts, but scales badly for large rank counts and ensembles of large rank counts.
This represents a major obstacle to large scale implementation.
Identified issues which are being analysed are (in rough priority order):
* [ ] https://github.com/MetOffice/lfric_core/pull/418
* [ ] #574
* [ ] https://github.com/MetOffice/lfric_core/issues/417
* [ ] #579
* [ ] #569
* [ ] [manage XIOS log file count](https://gitlab.in2p3.fr/ipsl/projets/xios-projects/xios/-/work_items/4)
* [ ] https://github.com/MetOffice/lfric_core/pull/419
* [ ] https://github.com/MetOffice/lfric_core/pull/421
* [ ] #577
* [ ] #578
* [ ] #580
## Validation
* use [branchHere](https://github.com/mo-marqh/lfric_apps) to evaluate calling counts and types using io_trace_intercept
* noting large performance overhead, don't trust nor compare timings
* once I/O patterns are validated, rerun tests to confirm actual timings
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.