MetOffice / MetOffice/lfric_core
Start dump and checkpoint dump need to be made more consistent and have better metadata support
- Dominant language
- Fortran
- Stars
- 26
- Forks
- 73
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 4
Description
### Are there any linked Issues or Pull Requests?
_No response_
### Brief description
The `lfric_atm` app was developed to read start dumps provided by `um2lfric`. These dumps have cell-centred data, and prior to the first timestep `lfric_atm` regrids/reshapes them to the native FEM form.
The checkpoint dump, however, contains raw data from the native `lfric_atm` fields
As time goes on, there is a growing need to start `lfric_atm` runs from another `lfric_atm` checkpoint dump.
1. Ideally, starting from "checkpoint-style" dumps will become the norm. It would be cleaner to do conversions of lfric2um dumps to "checkpoint-style" dumps as a separate step as the "checkpoint-style" dump format will in time converge to be the same or very similar to the diagnostic file format.
2. Checkpoint dumps currently contain checkpoints of ancil fields whether or not they are time varying. There is a strong case for simply re-reading ancil data for fixed fields whether for checkpoint restart or for starting a new run. There is also a case for re-reading and re-interpolating ancil data for time-varying ancils, particularly when starting a new run. This is because the ancil files or the interpolation requirements might be different in the new run. In other words, the model would potentially never write data derived from ancil fields to the checkpoint dump, and both new runs and continuation runs would read and interpolate data from the ancil files.
### A technical issue
Currently, the file definition for reading and writing the checkpoint dump is the same. There are some cases where we do not want this to be the case, but having different requirements is poorly supported:
1. Where some science sections are not run every time-step there may not be any data to output for some fields. For example, where the radiation runs every two timesteps, a warning is issued if, for example, checkpointing the run after timestep 3, stating the next timestep needs to be a radiation timestep. The run cannot be restarted as timestep 4 as timestep 4 is not a radiation timestep.
2. Some of the model state exists as simple scalars or arrays output to the checkpoint dump. For example, the random seed and various stochastic physics variables. A user may be constructing a start dump from other data and does not want to create dummy data for these items. Also, a user taking a dump from another configuration may not want to restart from whatever data was set for these data. Currently, the dump used for a model starting from a checkpoint style dump must include these data even if they are discarded or ignored due to logic in the model setup.
### Further details of the issue.
Some background on where we are currently. The current dumps are designed to mirror what happens in the UM. When a checkpoint is written, it contains ALL information required to restart the model. This is exactly what happens on a continuation run (`checkpoint_read=.true.`) - no other data (e.g. ancils) are re-read, as the checkpoint contains everything that is needed. When starting a new run from an LFRic dump (`init_option_checkpoint_dump`), only the model prognostics are required from this dump, new ancillary information is read and used.
This approach has certain advantages, which likely explain why it has been the accepted norm in the UM for so long:
- simplicity - the contents of the dumps are easily defined
- practicality - we don't necessarily know at the point at which a dump is written what it is going to be used for (a continuation run or a new run). Hence it contains all information to support both options.
- safety - when doing a continuation run, we are guaranteed to get the same ancillary data as would have been used if the checkpoint-restart hadn't happened - there is no risk or reading a different file, or time-interpolating to a different date.
- user-friendly - new runs can be started using only a dump, i.e. without reading any extra ancils (n.b. not sure the logic would currently allow this in LFRic, but it wouldn't be difficult to make work. It does work in the UM).
However, recently a number of issues have been raised over whether this is really the best long-term approach. In particular, the question has been raised whether we can avoid writing the ancillaries to the dumps:
- dumps are expensive to write from the operational suite, both in terms of computational cost and size on disk. The vast majority of the dump is taken up with writing the aerosol ancillary fields - 23 3D fields, which constitute more data than the rest of the prognostics combined! It would benefit the OS for these fields to not be in the dump.
- lfric2lfric is currently very memory constrained (because xios has to read all fields in the source and target dumps simultaneously), and therefore again it would be beneficial if it could work on fewer fields (i.e. not the ancillaries) until such time as it can work field-by-field and is not so memory constained
Removing the ancillaries from the dumps does have both technical and design challenges. It would need to overcome several of the key points above:
- we would need some way of re-reading and re-calculating the ancillaries for a continuation run. For non-time-updating ancils, this would require rewinding the xios clock to the original start of the run, to allow time-interpolation of the ancillary fields to this time, rather than the current time. For time-updating ancils, we would need to rewind the clock to the last update of the ancillary (which would likely be different for each ancillary).
- alternatively, we could keep the ancillaries in the dump for a continuation run, but lose them for a new run. This would require pre-defining what the dump is going to be used for when writing it. It would also require the contents of the dump to be different for different purposes. We would need some way of identifying whether a field is an ancil or not, i.e. can it be neglected from the checkpointing or not. This could be added to the fieldspec.
It's also not immediately clear which option is more computationally performant for continuation runs. In general, reading the ancils from their native files requires reading more data than reading them from the checkpoint file. This is because we need to read at least 2 time-points of data from the native files to time-interpolate them to the start time. Whereas once this has been done, we only have 1 time-point written and read from the dump.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the checkpoint dump read/write definitions and the init_option_checkpoint_dump path. Review how ancillary fields, science-section fields, scalars, and arrays are currently handled; done requires an agreed design that supports consistent checkpoint starts, continuation restarts, and improved metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100