NCAS-CMS / NCAS-CMS/cf-python

CFA Virtualisation using CMIP6 example data: Unable to aggregate

Open
#793 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
150
Forks
23
Avg merge
1d 11h
Merged PRs (30d)
2

Description

Example CMIP6 data (JASMIN)

files = [
    '/badc/cmip6/data/CMIP6/ScenarioMIP/CNRM-CERFACS/CNRM-ESM2-1/ssp119/r1i1p1f2/3hr/huss/gr/v20190328/huss_3hr_CNRM-ESM2-1_ssp119_r1i1p1f2_gr_201501010300-203501010000.nc',
    '/badc/cmip6/data/CMIP6/ScenarioMIP/CNRM-CERFACS/CNRM-ESM2-1/ssp119/r1i1p1f2/3hr/huss/gr/v20190328/huss_3hr_CNRM-ESM2-1_ssp119_r1i1p1f2_gr_203501010300-205501010000.nc',
    '/badc/cmip6/data/CMIP6/ScenarioMIP/CNRM-CERFACS/CNRM-ESM2-1/ssp119/r1i1p1f2/3hr/huss/gr/v20190328/huss_3hr_CNRM-ESM2-1_ssp119_r1i1p1f2_gr_205501010300-207501010000.nc',
    '/badc/cmip6/data/CMIP6/ScenarioMIP/CNRM-CERFACS/CNRM-ESM2-1/ssp119/r1i1p1f2/3hr/huss/gr/v20190328/huss_3hr_CNRM-ESM2-1_ssp119_r1i1p1f2_gr_207501010300-209501010000.nc',
    '/badc/cmip6/data/CMIP6/ScenarioMIP/CNRM-CERFACS/CNRM-ESM2-1/ssp119/r1i1p1f2/3hr/huss/gr/v20190328/huss_3hr_CNRM-ESM2-1_ssp119_r1i1p1f2_gr_209501010300-210101010000.nc'
]

Attempted to aggregate the first two example files (successful)

f = cf.read(files)
g = cf.aggregate(f[:2])

Normal cf.write functions properly here by creating a combined netCDF file of both files, but using with cfa=True results in one of the following, depending on if I take the whole of both Fields (116880 time steps):

RuntimeError: NetCDF: HDF error

or a subselection of the last 10 from file 1 and the first 10 from file 2 .

g = cf.aggregate([ f[0][-10:], f[1][:10] ])

File "/home/users/dwest77/Documents/cfa_python_dw/cf_dw/cf_python/cf/read_write/netcdf/netcdfwrite.py", line 106, in _write_as_cfa
    raise ValueError(
ValueError: Can't write <CF Field: specific_humidity(time(20), latitude(128), longitude(256)) 1> as a CFA-netCDF aggregation variable. Consider setting cfa={'strict': False}

cf-python 3.16.2 (latest)
cfdm 1.11.1.0 (latest)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the two examples with cf.read, cf.aggregate, and cf.write(cfa=True), then inspect cf/read_write/netcdf/netcdfwrite.py at _write_as_cfa. Compare the full and boundary-subset cases, and use the reported HDF error and strict-mode ValueError to trace the failing CFA-netCDF path. Done means the examples can be written with cfa=True without those failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.