Leung dust scheme causes CAM7 runs to fail with 1 task
- Dominant language
- Fortran
- Stars
- 352
- Forks
- 361
- Avg merge
- 6d 6h
- Merged PRs (30d)
- 8
Description
### Brief summary of bug
Running our coarsest CAM7 SE grid with 1 task on izumi/GNU results in a segfault very early in the run (before anything is logged to the atm logfile).
### General bug information
**CTSM version you are using:** ctsm5.3.024
**Does this bug cause significantly incorrect results in the model's science?** No
**Configurations affected:** CAM7 configurations (which are the ones that use the Leung dust scheme)
### Details of bug
We AMP SEs often run with our coarsest grids on a single task for testing, so this is impacting our ability to do that (and to use the TotalView debugger on izumi). Unfortunately, I cannot easily run a test with NAG because there's currently a known issue with running RRTMGP with NAG.
As described below, everything works OK on derecho, so it looks to be a memory issue.
@fvitt this might be relevant to you as well?
### Important details of your setup / configuration so we can reproduce the bug
Here are the tests I have run:
| compset | compiler| machine | resolution | misc config | dust scheme | result |
| --------| --------|---------|------------|-------------|-------------|---------|
| FLTHIST | gnu | izumi | ne3pg3 |./xmlchange NTASKS=1| Leung_2023 | FAIL |
| FLTHIST | gnu | izumi | ne3pg3 |**./xmlchange NTASKS=2**| Leung_2023 | PASS |
| FLTHIST | gnu | izumi | ne3pg3 |./xmlchange NTASKS=1| **Zender_2003** | PASS |
| FLTHIST | gnu | **derecho** | ne3pg3 |./xmlchange NTASKS=1| Leung_2023| PASS |
| FLTHIST | **intel** | **derecho** | ne3pg3 |./xmlchange NTASKS=1| Leung_2023| PASS |
### Important output or errors that show the problem
Error in cesm.log:
```
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 1536870 RUNNING AT i018.cgd.ucar.edu
= EXIT CODE: 139
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
```
Last log in lnd.log:
```
(GETFIL): attempting to find local file
surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc
(GETFIL): using /fs/cgd/csm/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc
Opened existing file /fs/cgd/csm/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc 152
ncd_inqvid: variable ETALAKE is not on dataset
WARNING:: ETALAKE not found on surface data set. All lake columns will have eta set equal to default value as a function of depth.
```
For context, here's what a successful run looks like in the lnd log:
```
(GETFIL): attempting to find local file
surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc
(GETFIL): using /fs/cgd/csm/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc
Opened existing file /fs/cgd/csm/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc 152
ncd_inqvid: variable ETALAKE is not on dataset
WARNING:: ETALAKE not found on surface data set. All lake columns will have eta set equal to default value as a function of depth.
ncd_inqvid: variable LAKEFETCH is not on dataset
WARNING:: LAKEFETCH not found on surface data set. All lake columns will have fetch set equal to default value as a function of depth.
Attempting to initialize time invariant variables for lakes
Successfully initialized time invariant variables for lakes
```
Contributor guide
Assessment
This issue has not been assessed yet.