pydata / pydata/xarray

`dtype` of `zarr` array unexpectedly changes when `fill_value` is specified

Open
#7,292 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-backends topic-zarr
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened?

Opening a zarr group which contains an array of integer dtype with a fill_value results in an xarray dataset in which the array has floating-point dtype.

What did you expect to happen?

An xarray dataset in which the array has the original integer dtype.

Minimal Complete Verifiable Example
import zarr
import xarray

#Create zarr with integer dtype and fill_value
grp = zarr.open_group("test.zarr")
arr = grp.create(shape=(10,), name="array", dtype="int8", fill_value=-1)
arr.attrs['_ARRAY_DIMENSIONS'] = ['dim1']

#Open in xarray to see that the dtype is now float32
ds = xarray.open_zarr("test.zarr", consolidated=False)
ds['array'].dtype
MVCE confirmation
  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
Relevant log output

No response

Anything else we need to know?

This is a result of https://github.com/pydata/xarray/issues/5475 where xarray's _FillValue has a different meaning to zarr's fill_value.

The change of dtype happens at https://github.com/pydata/xarray/blob/3c98ec7d96cc4b46664850cc7a40af2bc184fea0/xarray/coding/variables.py#L204 where xarray is trying to find a dtype where fill_value can represent "missing" data, wheras in zarr, fill_value can be any data value as its intent is to fill in missing chunks not represent missing data.

I'm not sure how best to fix this - maybe if the zarr fill value is clearly a non-missing value for the dtype then xarray should act as if it doesn't have a fill value? Happy to work on a PR if that seems to be a valid approach, although others may have thought on if that is a breaking change for some folks.

Environment
INSTALLED VERSIONS ------------------ commit: None python: 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] python-bits: 64 OS: Linux OS-release: 5.15.0-47-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: ('en_GB', 'UTF-8') libhdf5: None libnetcdf: None

xarray: 2022.11.0
pandas: 1.3.5
numpy: 1.21.6
scipy: 1.9.3
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.13.3
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2022.01.0
distributed: 2022.01.0
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2022.10.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 59.6.0
pip: 22.0.2
conda: None
pytest: None
IPython: None
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the MVCE with zarr and xarray, then inspect xarray/coding/variables.py around line 204 and the open_zarr path. Determine how zarr's fill_value should affect dtype without treating it as missing data; done means integer arrays retain their original dtype while the existing missing-value behavior remains covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.