Failures in GEOSCHEM/HEMCO configurations with different ocean & atm grids

### What happened?

In doing some testing with the exchange grid (setting `aoflux_grid = "xgrid"` in `user_nl_cpl`, I got a failure in `SMS_D_Ln9.f09_f09_mg17.FCnudged_GC.derecho_intel.cam-outfrq9s`:

```
dec2455.hsn.de.hpc.ucar.edu 484: forrtl: error (73): floating divide by zero
dec2455.hsn.de.hpc.ucar.edu 484: Image PC Routine Line Source
dec2455.hsn.de.hpc.ucar.edu 484: libpthread-2.31.s 0000149FBD5108C0 Unknown Unknown Unknown
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 000000000315AECC drydep_mod_mp_adu 4108 drydep_mod.F90
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 000000000313FCB0 drydep_mod_mp_dep 1774 drydep_mod.F90
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 000000000311CE7A drydep_mod_mp_do_ 316 drydep_mod.F90
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 0000000002C89ECB chemistry_mp_chem 3514 chemistry.F90
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 00000000012ACFAC physpkg_mp_tphysa 1604 physpkg.F90
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 00000000012A7F77 physpkg_mp_phys_r 1284 physpkg.F90
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 00000000009A9FA1 cam_comp_mp_cam_r 290 cam_comp.F90
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 0000000000959FFF atm_comp_nuopc_mp 1136 atm_comp_nuopc.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6533D86 execute 377 ESMCI_MethodTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6533932 execute 563 ESMCI_MethodTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC653352A c_esmc_methodtabl 317 ESMCI_MethodTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC68D388B esmf_attachmethod 1287 ESMF_AttachMethods.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC80493FD Unknown Unknown Unknown
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC618F279 callVFuncPtr 2167 ESMCI_FTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC618E2B8 ESMCI_FTableCallE 824 ESMCI_FTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC662BAB2 enter 2501 ESMCI_VMKernel.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6614346 enter 1216 ESMCI_VM.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC618F65F c_esmc_ftablecall 981 ESMCI_FTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6C134FC esmf_compmod_mp_e 1252 ESMF_Comp.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC74E3D6A esmf_gridcompmod_ 1903 ESMF_GridComp.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC7F85B75 nuopc_driver_mp_r 3694 NUOPC_Driver.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC7F8BDFA nuopc_driver_mp_e 3940 NUOPC_Driver.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6533D86 execute 377 ESMCI_MethodTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6533932 execute 563 ESMCI_MethodTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC653352A c_esmc_methodtabl 317 ESMCI_MethodTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC68D388B esmf_attachmethod 1287 ESMF_AttachMethods.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC7F83B76 nuopc_driver_mp_r 3615 NUOPC_Driver.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC618F279 callVFuncPtr 2167 ESMCI_FTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC618E2B8 ESMCI_FTableCallE 824 ESMCI_FTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC662BAB2 enter 2501 ESMCI_VMKernel.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6614346 enter 1216 ESMCI_VM.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC618F65F c_esmc_ftablecall 981 ESMCI_FTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6C134FC esmf_compmod_mp_e 1252 ESMF_Comp.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC74E3D6A esmf_gridcompmod_ 1903 ESMF_GridComp.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC7F85B75 nuopc_driver_mp_r 3694 NUOPC_Driver.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC7F8BDFA nuopc_driver_mp_e 3940 NUOPC_Driver.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6533D86 execute 377 ESMCI_MethodTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6533932 execute 563 ESMCI_MethodTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC653352A c_esmc_methodtabl 317 ESMCI_MethodTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC68D388B esmf_attachmethod 1287 ESMF_AttachMethods.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC7F83B76 nuopc_driver_mp_r 3615 NUOPC_Driver.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC618F279 callVFuncPtr 2167 ESMCI_FTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC618E2B8 ESMCI_FTableCallE 824 ESMCI_FTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC662BAB2 enter 2501 ESMCI_VMKernel.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6614346 enter 1216 ESMCI_VM.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC618F65F c_esmc_ftablecall 981 ESMCI_FTable.C
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC6C134FC esmf_compmod_mp_e 1252 ESMF_Comp.F90
dec2455.hsn.de.hpc.ucar.edu 484: libesmf.so 0000149FC74E3D6A esmf_gridcompmod_ 1903 ESMF_GridComp.F90
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 000000000044E467 MAIN__ 141 esmApp.F90
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 0000000000425D7D Unknown Unknown Unknown
dec2455.hsn.de.hpc.ucar.edu 484: libc-2.31.so 0000149FB8E0229D __libc_start_main Unknown Unknown
dec2455.hsn.de.hpc.ucar.edu 484: cesm.exe 0000000000425CAA Unknown Unknown Unknown
```

I also reproduced this failure in a run with the default of `aoflux_grid = "ogrid"` but when running with different ocean & atm grids: `SMS_D_Ln9.f09_g17.FCnudged_GC.derecho_intel.cam-outfrq9s`.

This is dying due to a divide by 0 in the geoschem drydep_mod, due to 0 values of ustar:

```Fortran
    ! surface resistance for particle
    RS   = 1.e0_f8 / (E0 * USTAR * (EB + EIM + EIN) * R1 )
```

It looks like the issue comes from the fact that, when the ocean grid differs from the atmosphere grid – either because these grids actually differ or because we're using the exchange grid (in which case there's an extra mapping done even if the atm & ocn grids are the same) – we get 0 values for ocean fields not only in land grid cells (which appear to be handled properly in CAM) but also in grid cells that are entirely sea ice-covered (or entirely sea ice + land).

Here is the relevant code for calculating USTAR in geoschem (in chemistry.F90):

```Fortran
! Field : USTAR
! Description: Friction velocity
! Unit : m/s
! Dimensions : nX, nY
! Note : We here combine the land friction velocity (fv) with
! the ocean friction velocity (ustar)
DO J = 1, nY
State_Met(LCHNK)%USTAR (1,J) = &
cam_in%fv(J) * ( cam_in%landFrac(J)) &
+ cam_in%uStar(J) * ( 1.0e+0_fp - cam_in%landFrac(J))
ENDDO
```

Notice that this code assumes that `cam_in%uStar` applies in the complement of landFrac, but in fact there is a third relevant fraction, the ice fraction, which is not accounted for here.

When I put in place a hack to workaround this ustar issue:

```diff
diff --git a/src/chemistry/geoschem/chemistry.F90 b/src/chemistry/geoschem/chemistry.F90
index ab56200c..2c64670b 100644
--- a/src/chemistry/geoschem/chemistry.F90
+++ b/src/chemistry/geoschem/chemistry.F90
@@ -2817,9 +2817,13 @@ subroutine chem_timestep_tend( state, ptend, cam_in, cam_out, dT, pbuf, fh2o )
! Note : We here combine the land friction velocity (fv) with
! the ocean friction velocity (ustar)
DO J = 1, nY
- State_Met(LCHNK)%USTAR (1,J) = &
- cam_in%fv(J) * ( cam_in%landFrac(J)) &
- + cam_in%uStar(J) * ( 1.0e+0_fp - cam_in%landFrac(J))
+ if (cam_in%uStar(J) == 0. .and. cam_in%landFrac(J) == 0.) then
+ State_Met(LCHNK)%USTAR(1,J) = 0.1
+ else
+ State_Met(LCHNK)%USTAR (1,J) = &
+ cam_in%fv(J) * ( cam_in%landFrac(J)) &
+ + cam_in%uStar(J) * ( 1.0e+0_fp - cam_in%landFrac(J))
+ end if
ENDDO

! Field : Z0
```

I got past the original crash, but got a new one here:

```
dec1643.hsn.de.hpc.ucar.edu 367: forrtl: error (73): floating divide by zero
dec1643.hsn.de.hpc.ucar.edu 367: Image PC Routine Line Source
dec1643.hsn.de.hpc.ucar.edu 367: libpthread-2.31.s 000014FBD087C8C0 Unknown Unknown Unknown
dec1643.hsn.de.hpc.ucar.edu 367: cesm.exe 0000000003126BE2 drydep_mod_mp_oce 705 drydep_mod.F90
dec1643.hsn.de.hpc.ucar.edu 367: cesm.exe 0000000003139B91 drydep_mod_mp_dep 1549 drydep_mod.F90
dec1643.hsn.de.hpc.ucar.edu 367: cesm.exe 000000000311DA7E drydep_mod_mp_do_ 316 drydep_mod.F90
dec1643.hsn.de.hpc.ucar.edu 367: cesm.exe 0000000002C8AACF chemistry_mp_chem 3518 chemistry.F90
dec1643.hsn.de.hpc.ucar.edu 367: cesm.exe 00000000012ACFAC physpkg_mp_tphysa 1604 physpkg.F90
dec1643.hsn.de.hpc.ucar.edu 367: cesm.exe 00000000012A7F77 physpkg_mp_phys_r 1284 physpkg.F90
dec1643.hsn.de.hpc.ucar.edu 367: cesm.exe 00000000009A9FA1 cam_comp_mp_cam_r 290 cam_comp.F90
dec1643.hsn.de.hpc.ucar.edu 367: cesm.exe 0000000000959FFF atm_comp_nuopc_mp 1136 atm_comp_nuopc.F90
```

That's dying due to a divide-by-zero of TEMPK in OCEANO3, which I think comes from

```Fortran
! Field : TSKIN
! Description: Surface skin temperature
! Remarks : NOT to be confused with TS (T at 2m) (hplin, 3/20/23)
! Unit : K
! Dimensions : nX, nY
State_Met(LCHNK)%TSKIN (1,:nY) = cam_in%SST(:nY)
```

which comes from `So_t`. Note that, like ustar, `atmExp_So_t` has 0 values over all-ice points in runs with different atm vs. ocn grids. (When the ocean is running on the same grid as the atmosphere, atmExp_So_t is 273.15 over land and doesn't seem to have patches of missing (or 273.15) values over sea ice points. But these 0 values appear when they're running on different grids.)

The 0 values come from the mapping from ocn to atm in the mediator, which uses a normalization of ofrac, which is the current ocean fraction (excluding both sea ice area and land area). @mvertens and I both feel that this ofrac normalization is the correct thing to do (and this was done in cpl7 as well as in CMEPS). Many fields (including "important" ones like the latent and sensible heat fluxes, etc.) are merged between ocn, ice and lnd before being passed to the atmosphere, so they are handled correctly. But a few fields – like ustar and So_t – are ocean-specific and so do not have this merge in the mediator. These fields need to be handled in CAM in a way that recognizes that they only apply over the ocean portion of the grid cell – not the sea ice portion or the land portion.

I see three options for the path forward, in order from least to most invasive in other components outside of CAM:

- Option 1: Change geoschem to be more similar to bulk_aero and modal_aero in this respect, doing its own calculations of ustar at least over sea ice, and possibly over ocean
- Option 2: Have sea ice send ustar, merge in CAM
- CAM's geoschem will then use sea ice's ustar in the merge.
- Note that bulk_aero and modal_aero could also be updated to use ustar from sea ice and ocean instead of calculating fv there.
- Option 3: Have sea ice send ustar, do the merge of ustar / fv in the mediator
- CAM's geoschem can be simplified to use this existing merged quantity everywhere
- Note that bulk_aero an modal_aero could also be updated to use this merged quantity

### What are the steps to reproduce the bug?

Run SMS_D_Ln9.f09_g17.FCnudged_GC.derecho_intel.cam-outfrq9s

### What CAM tag were you using?

cam6_4_032 in cesm3_0_alpha03c

### What machine were you running CAM on?

CISL machine (e.g. cheyenne)

### What compiler were you using?

Intel

### Path to a case directory, if applicable

/glade/derecho/scratch/sacks/SMS_D_Ln9.f09_g17.FCnudged_GC.derecho_intel.cam-outfrq9s.20241011_100351_kob7r6

### Will you be addressing this bug yourself?

No

### Extra info

For the failing case in /glade/derecho/scratch/sacks/SMS_D_Ln9.f09_g17.FCnudged_GC.derecho_intel.cam-outfrq9s.20241011_100351_kob7r6, I ran it twice. The first time it failed as noted above. The second time I introduced changes to get around the ustar issue:

```diff
diff --git a/src/chemistry/geoschem/chemistry.F90 b/src/chemistry/geoschem/chemistry.F90
index ab56200c..2c64670b 100644
--- a/src/chemistry/geoschem/chemistry.F90
+++ b/src/chemistry/geoschem/chemistry.F90
@@ -2817,9 +2817,13 @@ subroutine chem_timestep_tend( state, ptend, cam_in, cam_out, dT, pbuf, fh2o )
! Note : We here combine the land friction velocity (fv) with
! the ocean friction velocity (ustar)
DO J = 1, nY
- State_Met(LCHNK)%USTAR (1,J) = &
- cam_in%fv(J) * ( cam_in%landFrac(J)) &
- + cam_in%uStar(J) * ( 1.0e+0_fp - cam_in%landFrac(J))
+ if (cam_in%uStar(J) == 0. .and. cam_in%landFrac(J) == 0.) then
+ State_Met(LCHNK)%USTAR(1,J) = 0.1
+ else
+ State_Met(LCHNK)%USTAR (1,J) = &
+ cam_in%fv(J) * ( cam_in%landFrac(J)) &
+ + cam_in%uStar(J) * ( 1.0e+0_fp - cam_in%landFrac(J))
+ end if
ENDDO

! Field : Z0
```

and it died differently, I think due to 0 values in SSTs. See the two different cesm log files for backtraces for these two errors.