ESCOMP / ESCOMP/CMEPS

Leverage BFBFLAG to use reproducible sums in place of ESMF_VMAllreduce

Open
#650 12 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Fortran
Stars
35
Forks
106
Avg merge
6d 5h
Merged PRs (30d)
4

Description

As pointed out by @Katetc - we have a number of places in CMEPS where we use ESMF_VMAllreduce to do global sums. She found this in the course of testing answer changes with changing processor counts for T compsets (CISM forced by DLND), and for that use case, the issue is the global sums in the SMB renormalization (she says that turning off SMB renormalization allows answers to be bit-for-bit when changing processor counts - i.e., glc_renormalize_smb). But, from a quick look through the CMEPS code, it looks like this also comes up in post_rof_mod (dealing with negative runoff) and in prep_atm_mod (enthalpy correction).

I'm not sure we want to go so far as to require that global sums always be bit-for-bit with changing processor count, since that could impact performance. (My impression is that this isn't actually a CESM-wide requirement, but some components have it as a requirement. I'm not sure I'm right about that, though.) But it seems like we should at least have an option for reproducible global sums if a flag is set, and we can then use that flag in our testing. We have a currently-deprecated BFBFLAG, and the easiest course may be to bring that flag back into operation and use it for this purpose.

So I propose:
- Use BFBFLAG to set a nuopc runtime variable that controls whether we do more expensive global reductions that are bit-for-bit across processor counts
- If this flag is set, we can use the reproducible sums in https://github.com/ESCOMP/CESM_share/blob/main/src/shr_reprosum_mod.F90 in place of ESMF_VMAllreduce

@briandobbins @mvertens - also looping you in here; I'd welcome your thoughts.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.