ESCOMP / ESCOMP/CTSM

Current issues with sparse grids with CTSM

Open
#3,099 12 comments 0 reactions 0 assignees View on GitHub
code health enhancement
Dominant language
Fortran
Stars
352
Forks
361
Avg merge
6d 6h
Merged PRs (30d)
8

Description

### Brief summary of bug

@adrifoster has run into several issues with sparse grids in CTSM. The regular PPE 400 point one is working fine, but she's run into problems with smaller ones for 9 sparse points, as well as an 18 gridcell regional simulation. I'll call these two: sparse-9, and nwt_region_18 respectively, below.

I'm going to create this issue here as a parent for sparse grid issues in CTSM and submodules, as well as a place to gather discussion around these issues.

### General bug information

**CTSM version you are using:**

**Does this bug cause significantly incorrect results in the model's science?** No

**Configurations affected:**

### Details of bug

The problems run into:
- CDEPS fails on reading in the ESMF Mesh file with too many processors -- the fail is hard to diagnose
- CMEPS fails similarly
- On restart sparse-9 fail with negative LW down running with 9 tasks but works with 1
- On restart nwt_region_18 fail with negative LW down running with 18 tasks but works with 1

### Important details of your setup / configuration so we can reproduce the bug

The negative LW down might be due to a NaN, as it dies with a floating point exception earlier when DEBUG=TRUE.

### Important output or errors that show the problem

Here is an example of the floating point exception issue with DEBUG=TRUE for alaska_region-18

cesm.log:
```
dec0666.hsn.de.hpc.ucar.edu 53: forrtl: severe (174): SIGSEGV, segmentation fault occurred
dec0666.hsn.de.hpc.ucar.edu 53: Image PC Routine Line Source
dec0666.hsn.de.hpc.ucar.edu 53: libpthread-2.31.s 00001540A6CC68C0 Unknown Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libpiof.so 00001540B105C7D0 piodarray_mp_writ Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: cesm.exe 00000000004CA6B7 med_io_mod_mp_med 1094 med_io_mod.F90
dec0666.hsn.de.hpc.ucar.edu 53: cesm.exe 000000000056FEAC med_phases_restar 357 med_phases_restart_mod.F90
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A247 _ZN5ESMCI11Method Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A870 c_esmc_methodtabl Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEDDC5CB esmf_attachmethod Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF6CCF59 nuopc_modelbase_m Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD884 _ZN5ESMCI6FTable1 Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD36E ESMCI_FTableCallE Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AECB1BC4 _ZN5ESMCI2VM5ente Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BE08E c_esmc_ftablecall Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEF4D900 esmf_compmod_mp_e Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF293951 esmf_gridcompmod_ Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66DF37 nuopc_driver_mp_r Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66D78A nuopc_driver_mp_e Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A247 _ZN5ESMCI11Method Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A0E5 c_esmc_methodtabl Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEDDC6AE esmf_attachmethod Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66A8A2 nuopc_driver_mp_r Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD884 _ZN5ESMCI6FTable1 Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD36E ESMCI_FTableCallE Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AECB1BC4 _ZN5ESMCI2VM5ente Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BE08E c_esmc_ftablecall Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEF4D900 esmf_compmod_mp_e Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF293951 esmf_gridcompmod_ Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66DF37 nuopc_driver_mp_r Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66D78A nuopc_driver_mp_e Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A247 _ZN5ESMCI11Method Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEC2A0E5 c_esmc_methodtabl Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEDDC6AE esmf_attachmethod Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF66A8A2 nuopc_driver_mp_r Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD884 _ZN5ESMCI6FTable1 Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BD36E ESMCI_FTableCallE Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AECB1BC4 _ZN5ESMCI2VM5ente Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AE9BE08E c_esmc_ftablecall Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AEF4D900 esmf_compmod_mp_e Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libesmf.so 00001540AF293951 esmf_gridcompmod_ Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: cesm.exe 0000000000433189 MAIN__ 141 esmApp.F90
dec0666.hsn.de.hpc.ucar.edu 53: cesm.exe 0000000000421FBD Unknown Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: libc-2.31.so 00001540A25B829D __libc_start_main Unknown Unknown
dec0666.hsn.de.hpc.ucar.edu 53: cesm.exe 0000000000421EEA Unknown

```

Definition of done:

- [ ] Issue in CMEPS to gracefully die with too many tasks for gridcells
- [ ] Issue in CMEPS to gracefully die with too many tasks for gridcells
- [ ] Test cases with ERP tests for different task counts
- [ ] Test ERP for the 400 point sparse grid case

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.