Current issues with sparse grids with CTSM
- Dominant language
- Fortran
- Stars
- 352
- Forks
- 361
- Avg merge
- 6d 6h
- Merged PRs (30d)
- 8
Description
### Brief summary of bug
@adrifoster has run into several issues with sparse grids in CTSM. The regular PPE 400 point one is working fine, but she's run into problems with smaller ones for 9 sparse points, as well as an 18 gridcell regional simulation. I'll call these two: sparse-9, and nwt_region_18 respectively, below.
I'm going to create this issue here as a parent for sparse grid issues in CTSM and submodules, as well as a place to gather discussion around these issues.
### General bug information
**CTSM version you are using:**
**Does this bug cause significantly incorrect results in the model's science?** No
**Configurations affected:**
### Details of bug
The problems run into:
- CDEPS fails on reading in the ESMF Mesh file with too many processors -- the fail is hard to diagnose
- CMEPS fails similarly
- On restart sparse-9 fail with negative LW down running with 9 tasks but works with 1
- On restart nwt_region_18 fail with negative LW down running with 18 tasks but works with 1
### Important details of your setup / configuration so we can reproduce the bug
The negative LW down might be due to a NaN, as it dies with a floating point exception earlier when DEBUG=TRUE.
### Important output or errors that show the problem
Here is an example of the floating point exception issue with DEBUG=TRUE for alaska_region-18
cesm.log:
```
dec0666.hsn.de.hpc.ucar.edu 53: forrtl: severe (174): SIGSEGV, segmentation fault occurred
dec0666.hsn.de.hpc.ucar.edu 53: Image PC Routine Line Source
dec0666.hsn.de.hpc.ucar.edu 53: libpthread-2.31.s 00001540A6CC68C0 Unknown Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libpiof.so 00001540B105C7D0 piodarray_mp_writ Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: cesm.exe 00000000004CA6B7 med_io_mod_mp_med 1094 med_io_mod.F90
dec0666.hsn.de.hpc.ucar.edu 53: cesm.exe 000000000056FEAC med_phases_restar 357 med_phases_restart_mod.F90
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A247 _ZN5ESMCI11Method Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A870 c_esmc_methodtabl Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEDDC5CB esmf_attachmethod Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF6CCF59 nuopc_modelbase_m Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD884 _ZN5ESMCI6FTable1 Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD36E ESMCI_FTableCallE Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AECB1BC4 _ZN5ESMCI2VM5ente Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BE08E c_esmc_ftablecall Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEF4D900 esmf_compmod_mp_e Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF293951 esmf_gridcompmod_ Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66DF37 nuopc_driver_mp_r Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66D78A nuopc_driver_mp_e Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A247 _ZN5ESMCI11Method Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A0E5 c_esmc_methodtabl Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEDDC6AE esmf_attachmethod Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66A8A2 nuopc_driver_mp_r Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD884 _ZN5ESMCI6FTable1 Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD36E ESMCI_FTableCallE Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AECB1BC4 _ZN5ESMCI2VM5ente Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BE08E c_esmc_ftablecall Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEF4D900 esmf_compmod_mp_e Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF293951 esmf_gridcompmod_ Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66DF37 nuopc_driver_mp_r Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66D78A nuopc_driver_mp_e Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A247 _ZN5ESMCI11Method Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A0E5 c_esmc_methodtabl Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEDDC6AE esmf_attachmethod Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66A8A2 nuopc_driver_mp_r Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD884 _ZN5ESMCI6FTable1 Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD36E ESMCI_FTableCallE Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AECB1BC4 _ZN5ESMCI2VM5ente Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BE08E c_esmc_ftablecall Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEF4D900 esmf_compmod_mp_e Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF293951 esmf_gridcompmod_ Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: cesm.exe 0000000000433189 MAIN__ 141 esmApp.F90
dec0666.hsn.de.hpc.ucar.edu 53: cesm.exe 0000000000421FBD Unknown Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libc-2.31.so 00001540A25B829D __libc_start_main Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: cesm.exe 0000000000421EEA Unknown
```
Definition of done:
- [ ] Issue in CMEPS to gracefully die with too many tasks for gridcells
- [ ] Issue in CMEPS to gracefully die with too many tasks for gridcells
- [ ] Test cases with ERP tests for different task counts
- [ ] Test ERP for the 400 point sparse grid case
Contributor guide
Assessment
This issue has not been assessed yet.