ESCOMP / ESCOMP/CTSM

Leung dust scheme causes CAM7 runs to fail with 1 task

Open
#2,962 1 comment 0 reactions 1 assignee Claimed by @ekluzek View on GitHub
b4b bug investigation
Dominant language
Fortran
Stars
352
Forks
361
Avg merge
6d 6h
Merged PRs (30d)
8

Description

### Brief summary of bug

Running our coarsest CAM7 SE grid with 1 task on izumi/GNU results in a segfault very early in the run (before anything is logged to the atm logfile).

### General bug information

**CTSM version you are using:** ctsm5.3.024

**Does this bug cause significantly incorrect results in the model's science?** No

**Configurations affected:** CAM7 configurations (which are the ones that use the Leung dust scheme)

### Details of bug

We AMP SEs often run with our coarsest grids on a single task for testing, so this is impacting our ability to do that (and to use the TotalView debugger on izumi). Unfortunately, I cannot easily run a test with NAG because there's currently a known issue with running RRTMGP with NAG.

As described below, everything works OK on derecho, so it looks to be a memory issue.

@fvitt this might be relevant to you as well?

### Important details of your setup / configuration so we can reproduce the bug

Here are the tests I have run:

| compset | compiler| machine | resolution | misc config | dust scheme | result |
| --------| --------|---------|------------|-------------|-------------|---------|
| FLTHIST | gnu | izumi | ne3pg3 |./xmlchange NTASKS=1| Leung_2023 | FAIL |
| FLTHIST | gnu | izumi | ne3pg3 |**./xmlchange NTASKS=2**| Leung_2023 | PASS |
| FLTHIST | gnu | izumi | ne3pg3 |./xmlchange NTASKS=1| **Zender_2003** | PASS |
| FLTHIST | gnu | **derecho** | ne3pg3 |./xmlchange NTASKS=1| Leung_2023| PASS |
| FLTHIST | **intel** | **derecho** | ne3pg3 |./xmlchange NTASKS=1| Leung_2023| PASS |

### Important output or errors that show the problem

Error in cesm.log:
```
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 1536870 RUNNING AT i018.cgd.ucar.edu
= EXIT CODE: 139
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
```

Last log in lnd.log:
```
(GETFIL): attempting to find local file
surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc
(GETFIL): using /fs/cgd/csm/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc
Opened existing file /fs/cgd/csm/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc 152
ncd_inqvid: variable ETALAKE is not on dataset
WARNING:: ETALAKE not found on surface data set. All lake columns will have eta set equal to default value as a function of depth.
```

For context, here's what a successful run looks like in the lnd log:
```
(GETFIL): attempting to find local file
surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc
(GETFIL): using /fs/cgd/csm/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc
Opened existing file /fs/cgd/csm/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_ne3np4.pg3_hist_1850_78pfts_c240908.nc 152
ncd_inqvid: variable ETALAKE is not on dataset
WARNING:: ETALAKE not found on surface data set. All lake columns will have eta set equal to default value as a function of depth.
ncd_inqvid: variable LAKEFETCH is not on dataset
WARNING:: LAKEFETCH not found on surface data set. All lake columns will have fetch set equal to default value as a function of depth.
Attempting to initialize time invariant variables for lakes
Successfully initialized time invariant variables for lakes

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.