pydata / pydata/xarray

to_zarr() failing after concatenating netcdfs with different time indexes

Open
#4,347 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-cftime topic-error reporting
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened:
After concatenating two NetCDF DataSets with different cftime.DateTimeNoLeap coordinates, attempting to write to a Zarr store with ds.to_zarr() fails with an OutOfBoundsDatetime exception.

What you expected to happen:
I expect to_zarr() to execute successfully.

Minimal Complete Verifiable Example:

import xarray as xr
import cftime
import pandas as pd

# open a generic CESM dataset containing a time_bnds variable
url = 'http://adss.apcc21.org/opendap/CMIP5DB/cmip5_daily_BT/pr_day_CESM1-BGC_rcp85_r1i1p1_20760101-21001231.nc'
ds = xr.open_dataset(url)

# create two new DataSets with different, overlapping time indexes.
ds2 = ds.sel(time=slice(None, cftime.DatetimeNoLeap(2076, 3, 1, 1, 0, 0, 0)))
ds3 = ds.sel(time=slice(None, cftime.DatetimeNoLeap(2076, 2, 1, 1, 0, 0, 0)))

# concatenate the two DataSets, using the default fillvalue
ds4 = xr.concat([ds2, ds3], dim=pd.Index(['ds2','ds3'], name='ds'))

# fails with OutOfBoundsDatetime exception
zs = ds4.to_zarr('/tmp/my_zarr.zarr')

Anything else we need to know?:
I believe the problem is related to the implicit NaN fillvalue used in concatenating the time_bnds variable. I arrived at the code while trying to produce a minimal example of a similar error, where a DataSet concatenation would fail with a SerializationError if I tried to concatenate multiple datasets containing time_bnds variables. In the above example and in my earlier troubleshooting in production code, removing time_bnds with ds = ds.drop('time_bnds') made the to_zarr() command work.

This is a common use case since time_bnds indexes are generated by CESM climate model output.

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.7 (default, Mar 23 2020, 22:36:06)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 5.3.0-1032-aws
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.1

xarray: 0.16.0
pandas: 1.0.5
numpy: 1.19.1
scipy: 1.5.0
netCDF4: 1.4.2
pydap: None
h5netcdf: 0.8.0
h5py: 2.10.0
Nio: None
zarr: 2.3.2
cftime: 1.2.1
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.20.0
distributed: 2.20.0
matplotlib: 3.2.2
cartopy: 0.17.0
seaborn: None
numbagg: None
pint: None
setuptools: 49.2.0.post20200714
pip: 20.1.1
conda: None
pytest: None
IPython: 7.16.1
sphinx: None
/home/ubuntu/a

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the minimal example with xr.concat and ds4.to_zarr(), keeping the time_bnds variable, and confirm the OutOfBoundsDatetime failure. Trace the serialization path used by to_zarr for the concatenated time_bnds data; done means the example writes successfully to Zarr without requiring ds.drop('time_bnds').

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.