ESMCI / ESMCI/cmip7-prep

CMIP7-Prep fails for CESM3 output on Derecho with "Killed" message

Open
#55 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
8
Avg merge
1d 16h
Merged PRs (30d)
15

Description

Full output:

```
INFO:cmip7_prep.cmor_driver:Looking for pattern: ['cam.h0a']
INFO:cmip7_prep.cmor_driver:latest monthly time series file is (PosixPath('/glade/derecho/scratch/nanr/archive/b.e30_alpha08o.B1850C_LTso.ne30_t232_wgx3.cmip7straw.002/atm/hist/month_1/b.e30_alpha08o.B1850C_LTso.ne30_t232_wgx3.cmip7straw.002.cam.h0i.auxiliary.002101-002301.nc'), 23, 1)
INFO:cmip7_prep.cmor_driver:Loading data request content atmos
/glade/work/jedwards/conda-envs/CMORDEV/lib/python3.13/site-packages/data_request_api/utilities/decorators.py:31: UserWarning: Parameter 'consolidate' of function 'get_transformed_content' has a default value, but this default is overridden by the value specified in the configuration file.
warnings.warn(
/glade/work/jedwards/conda-envs/CMORDEV/lib/python3.13/site-packages/data_request_api/utilities/decorators.py:31: UserWarning: Parameter 'export' of function 'get_transformed_content' has a default value, but this default is overridden by the value specified in the configuration file.
warnings.warn(
INFO:cmip7_prep.cmor_driver:Content dictionary obtained
WARNING:root:Output of the current filtering request does not include all the BCV variables.
INFO:cmip7_prep.cmor_driver:CMORIZING 311 variables
INFO:cmip7_prep.mapping_compat:mapping file is cesm_to_cmip7.yaml
INFO:cmip7_prep.cmor_driver:Found 984 candidate timeseries files matching '*cam.h0a*.nc'
WARNING:cmip7_prep.pipeline:Skipping 'ccldncl_tavg-u-hxy-ccl': no mapping found in /glade/u/home/ccummins/cmip7-prep/data/cesm_to_cmip7.yaml
INFO:cmip7_prep.cmor_driver:============================================================
WARNING:cmip7_prep.cmor_driver:No timeseries files found for variable ccldncl_tavg-u-hxy-ccl
WARNING:cmip7_prep.pipeline:Skipping 'albisccp_tavg-u-hxy-cl': no mapping found in /glade/u/home/ccummins/cmip7-prep/data/cesm_to_cmip7.yaml
INFO:cmip7_prep.cmor_driver:============================================================
WARNING:cmip7_prep.cmor_driver:No timeseries files found for variable albisccp_tavg-u-hxy-cl
WARNING:cmip7_prep.pipeline:Skipping 'albisccp_tavg-u-hxy-cl': no mapping found in /glade/u/home/ccummins/cmip7-prep/data/cesm_to_cmip7.yaml
INFO:cmip7_prep.cmor_driver:============================================================
WARNING:cmip7_prep.cmor_driver:No timeseries files found for variable albisccp_tavg-u-hxy-cl
WARNING:cmip7_prep.pipeline:Skipping 'aod550volso4_tavg-u-hxy-u': no mapping found in /glade/u/home/ccummins/cmip7-prep/data/cesm_to_cmip7.yaml
INFO:cmip7_prep.cmor_driver:============================================================
WARNING:cmip7_prep.cmor_driver:No timeseries files found for variable aod550volso4_tavg-u-hxy-u
WARNING:cmip7_prep.pipeline:Skipping 'bldep_tavg-u-hxy-u': no mapping found in /glade/u/home/ccummins/cmip7-prep/data/cesm_to_cmip7.yaml
INFO:cmip7_prep.cmor_driver:============================================================
WARNING:cmip7_prep.cmor_driver:No timeseries files found for variable bldep_tavg-u-hxy-u
WARNING:cmip7_prep.pipeline:Skipping 'ccb_tavg-u-hxy-ccl': no mapping found in /glade/u/home/ccummins/cmip7-prep/data/cesm_to_cmip7.yaml
INFO:cmip7_prep.cmor_driver:============================================================
WARNING:cmip7_prep.cmor_driver:No timeseries files found for variable ccb_tavg-u-hxy-ccl
WARNING:cmip7_prep.pipeline:Skipping 'cct_tavg-u-hxy-ccl': no mapping found in /glade/u/home/ccummins/cmip7-prep/data/cesm_to_cmip7.yaml
INFO:cmip7_prep.cmor_driver:============================================================
WARNING:cmip7_prep.cmor_driver:No timeseries files found for variable cct_tavg-u-hxy-ccl
WARNING:cmip7_prep.pipeline:Skipping 'ci_tavg-u-hxy-u': no mapping found in /glade/u/home/ccummins/cmip7-prep/data/cesm_to_cmip7.yaml
INFO:cmip7_prep.cmor_driver:============================================================
WARNING:cmip7_prep.cmor_driver:No timeseries files found for variable ci_tavg-u-hxy-u
INFO:cmip7_prep.cmor_driver:============================================================
INFO:cmip7_prep.cmor_driver:Found 6 timeseries files for variable cl_tavg-al-hxy-u (model vars: ['CLOUD', 'P0', 'PS', 'TLAT', 'area', 'hyai', 'hyam', 'hybi', 'hybm', 'ilev', 'landfrac', 'landmask'])
INFO:cmip7_prep.cmor_driver:Starting processing for variable: cl_tavg-al-hxy-u
INFO:cmip7_prep.cmor_driver:Processing cl_tavg-al-hxy-u with dims ['time', 'lev', 'lat', 'lon']
INFO:cmip7_prep.cmor_driver:Opening native data for variable cl_tavg-al-hxy-u
INFO:cmip7_prep.mapping_compat:Required model vars to realize are ['CLOUD']
INFO:cmip7_prep.mapping_compat:Applying formula: CLOUD * 100 for name cl_tavg-al-hxy-u with model_vars ['CLOUD']
INFO:cmip7_prep.pipeline:Realized variable 'cl_tavg-al-hxy-u' with dims: ('time', 'lev', 'ncol')
INFO:cmip7_prep.regrid:Regridding cl_tavg-al-hxy-u using conservative map: /glade/campaign/cesm/cesmdata/inputdata/cpl/gridmaps/ne30pg3/map_ne30pg3_to_1x1d_aave.nc
INFO:cmip7_prep.regrid:Regridding complete. out_norms dims: ('time', 'lev', 'lat', 'lon')
INFO:cmip7_prep.regrid:Finished regridding var cl_tavg-al-hxy-u
INFO:cmip7_prep.regrid:Regridding PS using conservative map: /glade/campaign/cesm/cesmdata/inputdata/cpl/gridmaps/ne30pg3/map_ne30pg3_to_1x1d_aave.nc
INFO:cmip7_prep.regrid:Regridding complete. out_norms dims: ('time', 'lat', 'lon')
INFO:cmip7_prep.regrid:Finished regridding var PS
INFO:cmip7_prep.cmor_writer:CMOR logfile: /glade/derecho/scratch/ccummins/cmip7prep_output/logs/cmor_20260520T183934Z_cl_tavg-al-hxy-u.log
INFO:cmip7_prep.cmor_writer:CMOR JSON dataset: /glade/u/home/ccummins/cmip7-prep/data/cmor_dataset.json
INFO:cmip7_prep.cmor_driver:Setting region: glb
INFO:cmip7_prep.cmor_driver:Writing CMOR variable cl_tavg-al-hxy-u with frequency mon
INFO:cmip7_prep.cmor_driver:Writing CMOR variable cl_tavg-al-hxy-u with dims ['time', 'lev', 'lat', 'lon']
INFO:cmip7_prep.cmor_writer:Using CMOR table key: /glade/derecho/scratch/jedwards/cmip7-prep/cmip7-cmor-tables atmos
INFO:cmip7_prep.cmor_writer:Loading CMOR table for key 'atmos': /glade/derecho/scratch/jedwards/cmip7-prep/cmip7-cmor-tables/tables/CMIP7_atmos.json
INFO:cmip7_prep.cmor_writer:Defining CMOR axes for variable: cl
INFO:cmip7_prep.cmor_writer:time units: days since 0001-01-01 00:00:00 calendar: noleap
INFO:cmip7_prep.cmor_writer:Writing CMOR variable 6
Killed
```
The program exits with no error traceback and no data produced. Log files under `/glade/derecho/scratch/ccummins/cmip7prep_output/logs` were created for both variables, but they are empty:
```
(CMORDEV) ccummins@dec2243:/glade/derecho/scratch/ccummins/cmip7prep_output/logs> ls -lh
total 0
-rw-r--r-- 1 ccummins ncar 0 May 20 11:53 cmor_20260520T175353Z_cl_tavg-al-hxy-u.log
-rw-r--r-- 1 ccummins ncar 0 May 20 12:39 cmor_20260520T183934Z_cl_tavg-al-hxy-u.log
```
I suspect this is either an error from a C-library like CMOR or possibly an environment issue.

## Steps to Reproduce:

Access a Derecho compute node in an interactive session using `qinteractive`

Load the conda environment:
```
module load conda
conda activate /glade/work/jedwards/conda-envs/CMORDEV
```

Clone and install the new version of cmip7-prep:
```
git clone git@github.com:ESMCI/cmip7-prep.git
cd cmip7-prep
pip install -e .
```
Run on sample data:
`python ~/cmip7-prep/scripts/cmor_driver.py --realm atmos --model cesm --outdir /glade/derecho/scratch/ccummins/cmip7prep_output/ --tsdir /glade/derecho/scratch/nanr/archive/b.e30_alpha08o.B1850C_LTso.ne30_t232_wgx3.cmip7straw.002/atm/hist/month_1/ --debug`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.