Dataset.to_zarr() method writes array data even when passed compute=False
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened:
When writing out a Zarr dataset from an xr.Dataset, I pass the "compute=False" flag to tell xarray to write out the metadata but not the array data. When I inspect the Zarr dataset in the file system, I find that the array data has been written.
What you expected to happen:
I expected to find the metadata for my Zarr in the file system, but not the array data.
Minimal Complete Verifiable Example:
import xarray as xr, numpy as np, pandas as pd
ds = xr.Dataset({"foo": [10,20,30,40,50,60,70,80,90]})
ds.to_zarr("/tmp/bug.zarr", compute=False, consolidated=True)
bash$ cd /tmp/bug.zarr/foo
bash$ ls -la
total 20
drwxr-x--- 2 user group 4096 Mar 26 16:05 .
drwxr-x--- 6 user group 4096 Mar 26 16:05 ..
-rw-r----- 1 user group 88 Mar 26 16:05 0 # <---- I don't believe this should be here
-rw-r----- 1 user group 312 Mar 26 16:05 .zarray
-rw-r----- 1 user group 48 Mar 26 16:05 .zattrs
Anything else we need to know?:
Environment:
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.8.7 (default, Dec 22 2020, 10:37:26)
[GCC 10.2.0]
python-bits: 64
OS: Linux
OS-release: 5.7.17-1rodete5-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: None
libnetcdf: None
xarray: 0.17.0
pandas: 1.1.2
numpy: 1.19.2
scipy: 1.5.2
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.6.1
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.30.0
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 50.1.0
pip: 20.2.4
conda: None
pytest: None
IPython: None
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Dataset.to_zarr() entry point and reproduce the minimal example using compute=False and consolidated=True. Done means the Zarr store contains metadata but does not contain the array data chunk shown under foo/0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100