Fortran + MPI application - Caliper does not aggregate output

Summary of the problem:
A minimal Fortran + MPI code throws the Caliper error MPI is already finalized. Cannot aggregate output at runtime.

How to reproduce:
I have modified the example Fortran code that ships with Caliper (v2.12.0-dev under examples/apps/fortran-example.f) to be a minimal MPI code. I build the code with gfortran 11.4.0 and OpenMPI 4.1.2:

program fortran_example
    use caliper_mod
    use iso_c_binding, ONLY : C_INT64_T
    use mpi

    implicit none

    type(ConfigManager)   :: mgr

    integer               :: i, count, argc, ierr
    integer(C_INT64_T)    :: loop_attribute, iter_attribute

    logical               :: ret
    character(len=:), allocatable :: errmsg
    character(len=256)    :: arg

    call mpi_init(ierr) ! MPI INIT

    ! (Optional) create a ConfigManager object to control profiling.
    ! Users can provide a configuration string (e.g., 'runtime-report')
    ! on the command line.
    mgr = ConfigManager_new()
    call mgr%set_default_parameter('aggregate_across_ranks', 'false')

!!!!! ... OMITTED FOR BREVITY

    ! End 'main'
    call cali_end_region('main')

    ! Compute and flush output for the ConfigManager profiles.
    call mgr%flush
    call ConfigManager_delete(mgr)

    call mpi_finalize(ierr) ! MPI_FINALIZE

  end program fortran_example

I set both CALI_CONFIG and CALI_SERVICES_ENABLE (though I realise CALI_CONFIG takes precedence and CALI_SERVICES_ENABLE is not strictly needed):

> echo $CALI_CONFIG
runtime-report,mpi-report
> echo $CALI_SERVICES_ENABLE
aggregate,event,mpi,mpireport,timer,mpiflush

Running the test code fails as follows:

> ./test_caliper
== CALIPER: (0): default: mpireport: MPI is already finalized. Cannot aggregate output.
== CALIPER: (0): runtime-report: mpireport: MPI is already finalized. Cannot aggregate output.
== CALIPER: (0): mpi-report: mpireport: MPI is already finalized. Cannot aggregate output.

However, if I unset the CALI_XXX environment variables and set the services programmatically, everything works as expected. I.e. I modify the code to use add() and remove the section that reads the command line arguments:

..
    mgr = ConfigManager_new()
    call mgr%set_default_parameter('aggregate_across_ranks', 'false')
    call mgr%add('runtime-report')
    call mgr%add('mpi-report')

    ! Start configured profiling channels
    call mgr%start
...

Followed by unsetting the environment variables:

unset CALI_CONFIG
unset CALI_SERVICES_ENABLE

When I run the code, the behaviour is correct:

 ./test_caliper
Path       Time (E) Time (I) Time % (E) Time % (I)
main       0.000067 0.000105   0.333531   0.524120
  init     0.000008 0.000008   0.042344   0.042344
  mainloop 0.000030 0.000030   0.148244   0.148244
Function     Count (min) Count (max) Time (min) Time (max) Time (avg) Time %
                       1           1   0.001643   0.001643   0.001643 96.864394
MPI_Comm_dup           1           1   0.000053   0.000053   0.000053  3.135606

Can you please shed some light on this? For Caliper to be a long-term solution for us, we need to be able to define profiling reports at runtime (via environment variables ideally), and currently we are restricted to compile time because we need to define everything in the code. Hopefully you are able to reproduce the issue given the example above and suggest a fix.

Start with examples/apps/fortran-example.f and reproduce the failure using CALI_CONFIG and CALI_SERVICES_ENABLE with the minimal Fortran + MPI program. Compare that lifecycle with the working ConfigManager add() setup, focusing on flush and MPI finalization; done means runtime environment configuration aggregates successfully without the MPI-finalized error.