pydata / pydata/xarray

Dataset.to_zarr() method writes array data even when passed compute=False

Open
#5,080 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-documentation
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened:
When writing out a Zarr dataset from an xr.Dataset, I pass the "compute=False" flag to tell xarray to write out the metadata but not the array data. When I inspect the Zarr dataset in the file system, I find that the array data has been written.

What you expected to happen:
I expected to find the metadata for my Zarr in the file system, but not the array data.

Minimal Complete Verifiable Example:

import xarray as xr, numpy as np, pandas as pd
ds = xr.Dataset({"foo": [10,20,30,40,50,60,70,80,90]})
ds.to_zarr("/tmp/bug.zarr", compute=False, consolidated=True)

bash$ cd /tmp/bug.zarr/foo
bash$ ls -la
total 20
drwxr-x--- 2 user group 4096 Mar 26 16:05 .
drwxr-x--- 6 user group 4096 Mar 26 16:05 ..
-rw-r----- 1 user group   88 Mar 26 16:05 0        # <---- I don't believe this should be here
-rw-r----- 1 user group  312 Mar 26 16:05 .zarray
-rw-r----- 1 user group   48 Mar 26 16:05 .zattrs

Anything else we need to know?:

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.8.7 (default, Dec 22 2020, 10:37:26)
[GCC 10.2.0]
python-bits: 64
OS: Linux
OS-release: 5.7.17-1rodete5-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: None
libnetcdf: None

xarray: 0.17.0
pandas: 1.1.2
numpy: 1.19.2
scipy: 1.5.2
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.6.1
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.30.0
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 50.1.0
pip: 20.2.4
conda: None
pytest: None
IPython: None
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Dataset.to_zarr() entry point and reproduce the minimal example using compute=False and consolidated=True. Done means the Zarr store contains metadata but does not contain the array data chunk shown under foo/0.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.