Leverage BFBFLAG to use reproducible sums in place of ESMF_VMAllreduce
- Dominant language
- Fortran
- Stars
- 35
- Forks
- 106
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 4
Description
As pointed out by @Katetc - we have a number of places in CMEPS where we use ESMF_VMAllreduce to do global sums. She found this in the course of testing answer changes with changing processor counts for T compsets (CISM forced by DLND), and for that use case, the issue is the global sums in the SMB renormalization (she says that turning off SMB renormalization allows answers to be bit-for-bit when changing processor counts - i.e., glc_renormalize_smb). But, from a quick look through the CMEPS code, it looks like this also comes up in post_rof_mod (dealing with negative runoff) and in prep_atm_mod (enthalpy correction).
I'm not sure we want to go so far as to require that global sums always be bit-for-bit with changing processor count, since that could impact performance. (My impression is that this isn't actually a CESM-wide requirement, but some components have it as a requirement. I'm not sure I'm right about that, though.) But it seems like we should at least have an option for reproducible global sums if a flag is set, and we can then use that flag in our testing. We have a currently-deprecated BFBFLAG, and the easiest course may be to bring that flag back into operation and use it for this purpose.
So I propose:
- Use BFBFLAG to set a nuopc runtime variable that controls whether we do more expensive global reductions that are bit-for-bit across processor counts
- If this flag is set, we can use the reproducible sums in https://github.com/ESCOMP/CESM_share/blob/main/src/shr_reprosum_mod.F90 in place of ESMF_VMAllreduce
@briandobbins @mvertens - also looping you in here; I'd welcome your thoughts.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.