pydata / pydata/xarray

to_zarr: can't extend null coordinate (error on subsequent read)

Open
#8,383 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-zarr
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened?

When attempting to extend an on-disk (Zarr-backed) xarray by extending a time dimension, .to_zarr(<name>,mode='a',append_dim='t') fails upon subsequently loading the array if the time dimension was initially null.

This means that the initial write to the on-disk zarr must be special-cased, since it is impossible to initialize the array with a size-zero dimension for later expansion.

What did you expect to happen?

I expected the 0→1 case to function in the same way as the 1→2 case, which does work.

Minimal Complete Verifiable Example
import xarray as xr
import numpy as np
ds0 = xr.Dataset(coords={'t' : np.array([],dtype=np.int64)})
print('ds0', ds0)
ds0.to_zarr('foo.zarr',mode='w'); # Works
ds1 = xr.Dataset({'foo' : ('t',np.array([0]))},
                 coords={'t' : np.array([0],dtype=np.int64)})
print('ds1',ds1)
ds1.to_zarr('foo.zarr',mode='a',append_dim='t'); # Completes successfully, but...
ds_load = xr.open_zarr('foo.zarr') # Errors
MVCE confirmation
  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
ds0 <xarray.Dataset>
Dimensions:  (t: 0)
Coordinates:
  * t        (t) int64 
Data variables:
    *empty*
ds1 <xarray.Dataset>
Dimensions:  (t: 1)
Coordinates:
  * t        (t) int64 0
Data variables:
    foo      (t) int64 0

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 10
      8 print('ds1',ds1)
      9 ds1.to_zarr('foo.zarr',mode='a',append_dim='t'); # Completes successfully, but...
---> 10 ds_load = xr.open_zarr('foo.zarr') # Errors

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py:900, in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, chunk_store, storage_options, decode_timedelta, use_cftime, zarr_version, chunked_array_type, from_array_kwargs, **kwargs)
    886     raise TypeError(
    887         "open_zarr() got unexpected keyword arguments " + ",".join(kwargs.keys())
    888     )
    890 backend_kwargs = {
    891     "synchronizer": synchronizer,
    892     "consolidated": consolidated,
   (...)
    897     "zarr_version": zarr_version,
    898 }
--> 900 ds = open_dataset(
    901     filename_or_obj=store,
    902     group=group,
    903     decode_cf=decode_cf,
    904     mask_and_scale=mask_and_scale,
    905     decode_times=decode_times,
    906     concat_characters=concat_characters,
    907     decode_coords=decode_coords,
    908     engine="zarr",
    909     chunks=chunks,
    910     drop_variables=drop_variables,
    911     chunked_array_type=chunked_array_type,
    912     from_array_kwargs=from_array_kwargs,
    913     backend_kwargs=backend_kwargs,
    914     decode_timedelta=decode_timedelta,
    915     use_cftime=use_cftime,
    916     zarr_version=zarr_version,
    917 )
    918 return ds

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/api.py:573, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
    561 decoders = _resolve_decoders_kwargs(
    562     decode_cf,
    563     open_backend_dataset_parameters=backend.open_dataset_parameters,
   (...)
    569     decode_coords=decode_coords,
    570 )
    572 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 573 backend_ds = backend.open_dataset(
    574     filename_or_obj,
    575     drop_variables=drop_variables,
    576     **decoders,
    577     **kwargs,
    578 )
    579 ds = _dataset_from_backend_dataset(
    580     backend_ds,
    581     filename_or_obj,
   (...)
    591     **kwargs,
    592 )
    593 return ds

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py:982, in ZarrBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, stacklevel, zarr_version)
    980 store_entrypoint = StoreBackendEntrypoint()
    981 with close_on_error(store):
--> 982     ds = store_entrypoint.open_dataset(
    983         store,
    984         mask_and_scale=mask_and_scale,
    985         decode_times=decode_times,
    986         concat_characters=concat_characters,
    987         decode_coords=decode_coords,
    988         drop_variables=drop_variables,
    989         use_cftime=use_cftime,
    990         decode_timedelta=decode_timedelta,
    991     )
    992 return ds

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/store.py:58, in StoreBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta)
     44 encoding = filename_or_obj.get_encoding()
     46 vars, attrs, coord_names = conventions.decode_cf_variables(
     47     vars,
     48     attrs,
   (...)
     55     decode_timedelta=decode_timedelta,
     56 )
---> 58 ds = Dataset(vars, attrs=attrs)
     59 ds = ds.set_coords(coord_names.intersection(vars))
     60 ds.set_close(filename_or_obj.close)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/dataset.py:696, in Dataset.__init__(self, data_vars, coords, attrs)
    693 if isinstance(coords, Dataset):
    694     coords = coords._variables
--> 696 variables, coord_names, dims, indexes, _ = merge_data_and_coords(
    697     data_vars, coords
    698 )
    700 self._attrs = dict(attrs) if attrs is not None else None
    701 self._close = None

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/dataset.py:425, in merge_data_and_coords(data_vars, coords)
    421     coords = create_coords_with_default_indexes(coords, data_vars)
    423 # exclude coords from alignment (all variables in a Coordinates object should
    424 # already be aligned together) and use coordinates' indexes to align data_vars
--> 425 return merge_core(
    426     [data_vars, coords],
    427     compat="broadcast_equals",
    428     join="outer",
    429     explicit_coords=tuple(coords),
    430     indexes=coords.xindexes,
    431     priority_arg=1,
    432     skip_align_args=[1],
    433 )

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/merge.py:724, in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value, skip_align_args)
    719 prioritized = _get_priority_vars_and_indexes(aligned, priority_arg, compat=compat)
    720 variables, out_indexes = merge_collected(
    721     collected, prioritized, compat=compat, combine_attrs=combine_attrs
    722 )
--> 724 dims = calculate_dimensions(variables)
    726 coord_names, noncoord_names = determine_coords(coerced)
    727 if compat == "minimal":
    728     # coordinates may be dropped in merged results

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/variable.py:2997, in calculate_dimensions(variables)
   2995             last_used[dim] = k
   2996         elif dims[dim] != size:
-> 2997             raise ValueError(
   2998                 f"conflicting sizes for dimension {dim!r}: "
   2999                 f"length {size} on {k!r} and length {dims[dim]} on {last_used!r}"
   3000             )
   3001 return dims

ValueError: conflicting sizes for dimension 't': length 1 on 't' and length 2 on {'t': 'foo'}
Anything else we need to know?

No response

Environment
INSTALLED VERSIONS ------------------ commit: None python: 3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:49:32) [GCC 12.3.0] python-bits: 64 OS: Linux OS-release: 5.15.0-79-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.9.1

xarray: 2023.10.1
pandas: 2.1.1
numpy: 1.24.4
scipy: 1.11.3
netCDF4: 1.6.3
pydap: installed
h5netcdf: 1.2.0
h5py: 3.8.0
Nio: 1.5.5
zarr: 2.16.1
cftime: 1.6.2
nc_time_axis: 1.4.1
PseudoNetCDF: None
iris: 3.4.1
bottleneck: 1.3.7
dask: 2023.10.0
distributed: 2023.10.0
matplotlib: 3.8.0
cartopy: 0.22.0
seaborn: 0.13.0
numbagg: 0.6.0
fsspec: 2023.10.0
cupy: None
pint: 0.22
sparse: 0.14.0
flox: None
numpy_groupies: None
setuptools: 68.2.2
pip: 23.3.1
conda: None
pytest: None
mypy: None
IPython: 8.16.1
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with xarray/backends/zarr.py, especially the to_zarr and open_zarr entry points, and reproduce the 0→1 append case from the issue. Trace how the dimension sizes are written and loaded; done means the example opens successfully after appending to an initially null coordinate, with a regression test covering that case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.