MetOffice / MetOffice/lfric_apps

Input Data parallelisation

Open
#575 1 comment 0 reactions 4 assignees Claimed by @EdHone View on GitHub
Dominant language
Fortran
Stars
31
Forks
118
Avg merge
2d 4h
Merged PRs (30d)
8

Description

Analyses of the interactions with parallel file systems at scale demonstrate that `lfric_atm` is not scaling well to large rank counts.

`lfric_atm` requires significant quantities of data to initialise the model, including configurations, mesh definitions, start dumps & ancillaries.

Some facets of the code base implement simple storage interaction patterns where each rank opens a file as it needs it and obtains the information it requires. This works fine for small rank counts and okay for medium rank counts, but scales badly for large rank counts and ensembles of large rank counts.

This represents a major obstacle to large scale implementation.

Identified issues which are being analysed are (in rough priority order):

* [ ] https://github.com/MetOffice/lfric_core/pull/418
* [ ] #574
* [ ] https://github.com/MetOffice/lfric_core/issues/417
* [ ] #579
* [ ] #569
* [ ] [manage XIOS log file count](https://gitlab.in2p3.fr/ipsl/projets/xios-projects/xios/-/work_items/4)
* [ ] https://github.com/MetOffice/lfric_core/pull/419
* [ ] https://github.com/MetOffice/lfric_core/pull/421
* [ ] #577
* [ ] #578
* [ ] #580

## Validation

* use [branchHere](https://github.com/mo-marqh/lfric_apps) to evaluate calling counts and types using io_trace_intercept
* noting large performance overhead, don't trust nor compare timings
* once I/O patterns are validated, rerun tests to confirm actual timings

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.