`dtype` of `zarr` array unexpectedly changes when `fill_value` is specified
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened?
Opening a zarr group which contains an array of integer dtype with a fill_value results in an xarray dataset in which the array has floating-point dtype.
What did you expect to happen?
An xarray dataset in which the array has the original integer dtype.
Minimal Complete Verifiable Example
import zarr
import xarray
#Create zarr with integer dtype and fill_value
grp = zarr.open_group("test.zarr")
arr = grp.create(shape=(10,), name="array", dtype="int8", fill_value=-1)
arr.attrs['_ARRAY_DIMENSIONS'] = ['dim1']
#Open in xarray to see that the dtype is now float32
ds = xarray.open_zarr("test.zarr", consolidated=False)
ds['array'].dtype
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
Relevant log output
No response
Anything else we need to know?
This is a result of https://github.com/pydata/xarray/issues/5475 where xarray's _FillValue has a different meaning to zarr's fill_value.
The change of dtype happens at https://github.com/pydata/xarray/blob/3c98ec7d96cc4b46664850cc7a40af2bc184fea0/xarray/coding/variables.py#L204 where xarray is trying to find a dtype where fill_value can represent "missing" data, wheras in zarr, fill_value can be any data value as its intent is to fill in missing chunks not represent missing data.
I'm not sure how best to fix this - maybe if the zarr fill value is clearly a non-missing value for the dtype then xarray should act as if it doesn't have a fill value? Happy to work on a PR if that seems to be a valid approach, although others may have thought on if that is a breaking change for some folks.
Environment
xarray: 2022.11.0
pandas: 1.3.5
numpy: 1.21.6
scipy: 1.9.3
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.13.3
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2022.01.0
distributed: 2022.01.0
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2022.10.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 59.6.0
pip: 22.0.2
conda: None
pytest: None
IPython: None
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the MVCE with zarr and xarray, then inspect xarray/coding/variables.py around line 204 and the open_zarr path. Determine how zarr's fill_value should affect dtype without treating it as missing data; done means integer arrays retain their original dtype while the existing missing-value behavior remains covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100