Round tripping Zarr datasets with scaling
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened?
Data was loaded into Xarray via OpenDataCube dc.load method. This Xarray was persisted to Zarr using
xx.time.encoding['units'] = "seconds since 1970-01-01 00:00:00"
xx.time.attrs = {}
xx.to_zarr('test.zarr',mode='w',consolidated=True)
When this zarr is loaded back using
ds_z = xr.open_dataset('test.zarr', engine = 'zarr', consolidated = True)
The round tripped dataset has scaling applied with encodings saved in bands e.g. ds_z.red.encoding as
ds_z.red.encoding
{'chunks': (1, 610, 522),
'preferred_chunks': {'time': 1, 'y': 610, 'x': 522},
'compressor': Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0),
'filters': None,
'scale_factor': 0.0001,
'add_offset': -0.1,
'dtype': dtype('uint16'),
'coordinates': 'spatial_ref'}
The values returned do not have the scaling auto-applied.
What did you expect to happen?
Scaling to DataArrays is auto-applied when rounding tripping to-from Zarr via Xarray.
Minimal Complete Verifiable Example
ds.time.encoding['units'] = "seconds since 1970-01-01 00:00:00"
ds.time.attrs = {}
ds.to_zarr('test.zarr',mode='w',consolidated=True)
ds_z = xr.open_dataset('test.zarr', engine = 'zarr', consolidated = True)
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
No response
Anything else we need to know?
No response
Environment
xarray: 2024.10.0
pandas: 2.2.3
numpy: 1.26.4
scipy: 1.14.1
netCDF4: 1.7.2
pydap: 3.5
h5netcdf: 1.4.0
h5py: 3.12.1
zarr: 2.18.3
cftime: 1.6.4.post1
nc_time_axis: 1.4.1
iris: None
bottleneck: 1.4.2
dask: 2024.7.1
distributed: 2024.7.1
matplotlib: 3.8.4
cartopy: 0.24.1
seaborn: 0.13.2
numbagg: 0.8.2
fsspec: 2024.10.0
cupy: None
pint: 0.24.4
sparse: 0.15.4
flox: 0.9.14
numpy_groupies: 0.11.2
setuptools: 75.3.0
pip: 24.3.1
conda: None
pytest: 8.3.3
mypy: None
IPython: 8.29.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the minimal example using to_zarr and xr.open_dataset with the supplied environment details, then inspect how the scale_factor and add_offset encodings are written and read. Confirm the expected decoded values after round-tripping and add coverage for the demonstrated scaling case; done means the reopened DataArray automatically applies the stored scaling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100