ESMCI / ESMCI/cmip7-prep

memory error on NIRD when processing daily data for NorESM

Open
#66 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
8
Avg merge
1d 16h
Merged PRs (30d)
15

Description

I am using the validation-script branch on nird.

Monthly processing using the following worked
`./cmor_driver.py --realm atmos --tsdir ~/test_cmor/n1850.ne16pg3_tn14.noresm3_0_beta17.Run12-1.restartfromold.2026-05-21/atm/timeseries --outdir ~/test_cmor/n1850.ne16pg3_tn14.noresm3_0_beta17.Run12-1.restartfromold.2026-05-21/cmor --workers 1 --resolution ne16 --model noresm `

However, following gives rise to a memory error
`./cmor_driver.py --realm atmos --tsdir ~/test_cmor/n1850.ne16pg3_tn14.noresm3_0_beta17.Run12-1.restartfromold.2026-05-21/atm/timeseries --outdir ~/test_cmor/n1850.ne16pg3_tn14.noresm3_0_beta17.Run12-1.restartfromold.2026-05-21/cmor --workers 1 --resolution ne16 --model noresm --frequency day`

ERROR:cmip7_prep.cmor_driver:Exception while processing zg_tavg-p39-hy-air with dims ['time', 'plev', 'lat']: MemoryError((58, 12, 96, 144), dtype('float64'))

zg is the last field to be processed - but there is a vertical interpolation that is being done.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with cmor_driver.py and reproduce the daily command on NIRD, comparing it with the working monthly command. Trace processing of zg_tavg-p39-hy-air and its vertical interpolation; done means daily processing completes without the reported MemoryError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.