pydata / pydata/xarray

Attributes encoding compatibility between backends

Open
#5,226 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-backends topic-zarr
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened:

Let's create an Zarr dataset with some "less common" dtype and fill value, open it with Xarray and save the dataset as NetCDF:

import xarray as xr
import zarr

g = zarr.group()
g.create('arr', shape=3, fill_value='z', dtype='<U1')
g['arr'].attrs['_ARRAY_DIMENSIONS'] = ('dim_1')

# -- without masking fill values
ds = xr.open_zarr(g.store, mask_and_scale=False)

ds.arr.attrs   # returns {'_FillValue': 'z'}

# error: netCDF4 does not yet support setting a fill value for variable-length strings
ds.to_netcdf('test.nc')

# -- with masking fill values
ds2 = xr.open_zarr(g.store, mask_and_scale=True)

# returns a dict that includes item _FillValue': 'z'
ds2.arr.encoding

# same error than above
ds2.to_netcdf('out2.nc')

What you expected to happen:

Seamless conversion (read/write) from one backend to another. Is there anything we could do to improve the case shown here above, and maybe other cases like the one described in #5223?

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
libhdf5: None
libnetcdf: None

xarray: 0.17.0
pandas: 1.0.3
numpy: 1.18.1
scipy: 1.3.1
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.8.1
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.11.0
distributed: 2.14.0
matplotlib: 3.1.1
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 46.1.3.post20200325
pip: 19.2.3
conda: None
pytest: 5.4.1
IPython: 7.13.0
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue using the shown zarr.group, xr.open_zarr, and Dataset.to_netcdf calls, with and without mask_and_scale. Trace how the Zarr _FillValue reaches attrs or encoding and is passed to the NetCDF backend; done means the demonstrated variable-length string dataset can be converted without the fill-value error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.