to_zarr: can't extend null coordinate (error on subsequent read)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened?
When attempting to extend an on-disk (Zarr-backed) xarray by extending a time dimension, .to_zarr(<name>,mode='a',append_dim='t') fails upon subsequently loading the array if the time dimension was initially null.
This means that the initial write to the on-disk zarr must be special-cased, since it is impossible to initialize the array with a size-zero dimension for later expansion.
What did you expect to happen?
I expected the 0→1 case to function in the same way as the 1→2 case, which does work.
Minimal Complete Verifiable Example
import xarray as xr
import numpy as np
ds0 = xr.Dataset(coords={'t' : np.array([],dtype=np.int64)})
print('ds0', ds0)
ds0.to_zarr('foo.zarr',mode='w'); # Works
ds1 = xr.Dataset({'foo' : ('t',np.array([0]))},
coords={'t' : np.array([0],dtype=np.int64)})
print('ds1',ds1)
ds1.to_zarr('foo.zarr',mode='a',append_dim='t'); # Completes successfully, but...
ds_load = xr.open_zarr('foo.zarr') # Errors
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
ds0 <xarray.Dataset>
Dimensions: (t: 0)
Coordinates:
* t (t) int64
Data variables:
*empty*
ds1 <xarray.Dataset>
Dimensions: (t: 1)
Coordinates:
* t (t) int64 0
Data variables:
foo (t) int64 0
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[2], line 10
8 print('ds1',ds1)
9 ds1.to_zarr('foo.zarr',mode='a',append_dim='t'); # Completes successfully, but...
---> 10 ds_load = xr.open_zarr('foo.zarr') # Errors
File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py:900, in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, chunk_store, storage_options, decode_timedelta, use_cftime, zarr_version, chunked_array_type, from_array_kwargs, **kwargs)
886 raise TypeError(
887 "open_zarr() got unexpected keyword arguments " + ",".join(kwargs.keys())
888 )
890 backend_kwargs = {
891 "synchronizer": synchronizer,
892 "consolidated": consolidated,
(...)
897 "zarr_version": zarr_version,
898 }
--> 900 ds = open_dataset(
901 filename_or_obj=store,
902 group=group,
903 decode_cf=decode_cf,
904 mask_and_scale=mask_and_scale,
905 decode_times=decode_times,
906 concat_characters=concat_characters,
907 decode_coords=decode_coords,
908 engine="zarr",
909 chunks=chunks,
910 drop_variables=drop_variables,
911 chunked_array_type=chunked_array_type,
912 from_array_kwargs=from_array_kwargs,
913 backend_kwargs=backend_kwargs,
914 decode_timedelta=decode_timedelta,
915 use_cftime=use_cftime,
916 zarr_version=zarr_version,
917 )
918 return ds
File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/api.py:573, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
561 decoders = _resolve_decoders_kwargs(
562 decode_cf,
563 open_backend_dataset_parameters=backend.open_dataset_parameters,
(...)
569 decode_coords=decode_coords,
570 )
572 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 573 backend_ds = backend.open_dataset(
574 filename_or_obj,
575 drop_variables=drop_variables,
576 **decoders,
577 **kwargs,
578 )
579 ds = _dataset_from_backend_dataset(
580 backend_ds,
581 filename_or_obj,
(...)
591 **kwargs,
592 )
593 return ds
File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py:982, in ZarrBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, stacklevel, zarr_version)
980 store_entrypoint = StoreBackendEntrypoint()
981 with close_on_error(store):
--> 982 ds = store_entrypoint.open_dataset(
983 store,
984 mask_and_scale=mask_and_scale,
985 decode_times=decode_times,
986 concat_characters=concat_characters,
987 decode_coords=decode_coords,
988 drop_variables=drop_variables,
989 use_cftime=use_cftime,
990 decode_timedelta=decode_timedelta,
991 )
992 return ds
File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/store.py:58, in StoreBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta)
44 encoding = filename_or_obj.get_encoding()
46 vars, attrs, coord_names = conventions.decode_cf_variables(
47 vars,
48 attrs,
(...)
55 decode_timedelta=decode_timedelta,
56 )
---> 58 ds = Dataset(vars, attrs=attrs)
59 ds = ds.set_coords(coord_names.intersection(vars))
60 ds.set_close(filename_or_obj.close)
File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/dataset.py:696, in Dataset.__init__(self, data_vars, coords, attrs)
693 if isinstance(coords, Dataset):
694 coords = coords._variables
--> 696 variables, coord_names, dims, indexes, _ = merge_data_and_coords(
697 data_vars, coords
698 )
700 self._attrs = dict(attrs) if attrs is not None else None
701 self._close = None
File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/dataset.py:425, in merge_data_and_coords(data_vars, coords)
421 coords = create_coords_with_default_indexes(coords, data_vars)
423 # exclude coords from alignment (all variables in a Coordinates object should
424 # already be aligned together) and use coordinates' indexes to align data_vars
--> 425 return merge_core(
426 [data_vars, coords],
427 compat="broadcast_equals",
428 join="outer",
429 explicit_coords=tuple(coords),
430 indexes=coords.xindexes,
431 priority_arg=1,
432 skip_align_args=[1],
433 )
File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/merge.py:724, in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value, skip_align_args)
719 prioritized = _get_priority_vars_and_indexes(aligned, priority_arg, compat=compat)
720 variables, out_indexes = merge_collected(
721 collected, prioritized, compat=compat, combine_attrs=combine_attrs
722 )
--> 724 dims = calculate_dimensions(variables)
726 coord_names, noncoord_names = determine_coords(coerced)
727 if compat == "minimal":
728 # coordinates may be dropped in merged results
File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/variable.py:2997, in calculate_dimensions(variables)
2995 last_used[dim] = k
2996 elif dims[dim] != size:
-> 2997 raise ValueError(
2998 f"conflicting sizes for dimension {dim!r}: "
2999 f"length {size} on {k!r} and length {dims[dim]} on {last_used!r}"
3000 )
3001 return dims
ValueError: conflicting sizes for dimension 't': length 1 on 't' and length 2 on {'t': 'foo'}
Anything else we need to know?
No response
Environment
xarray: 2023.10.1
pandas: 2.1.1
numpy: 1.24.4
scipy: 1.11.3
netCDF4: 1.6.3
pydap: installed
h5netcdf: 1.2.0
h5py: 3.8.0
Nio: 1.5.5
zarr: 2.16.1
cftime: 1.6.2
nc_time_axis: 1.4.1
PseudoNetCDF: None
iris: 3.4.1
bottleneck: 1.3.7
dask: 2023.10.0
distributed: 2023.10.0
matplotlib: 3.8.0
cartopy: 0.22.0
seaborn: 0.13.0
numbagg: 0.6.0
fsspec: 2023.10.0
cupy: None
pint: 0.22
sparse: 0.14.0
flox: None
numpy_groupies: None
setuptools: 68.2.2
pip: 23.3.1
conda: None
pytest: None
mypy: None
IPython: 8.16.1
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with xarray/backends/zarr.py, especially the to_zarr and open_zarr entry points, and reproduce the 0→1 append case from the issue. Trace how the dimension sizes are written and loaded; done means the example opens successfully after appending to an initially null coordinate, with a regression test covering that case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100