pydata / pydata/xarray

`Dataset.to_zarr` compute=False should allow access to awaitable

Open
#6,383 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement topic-dask topic-zarr
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened?

I have xarray, zarr installed, but not dask, and am trying to call to_zarr in an async routine. I am looking for something I can await. The doc claims that a Dask.Delayed is returned. I understand that if I have a dask client open with asynchronous=True I can await the result.

However, not using Dask. Is there some way to get an awaitable from this object without a dask client?

What did you expect to happen?

I should get something back I can await in my async routine.

Minimal Complete Verifiable Example
import xarray as xr
from asyncio import get_event_loop

ds = xr.Dataset(data_vars = dict(x = ('x', [1, 2])))
deld = ds.to_zarr("bar.zarr", compute=False)
loop.run_until_complete(deld. ...?)
Relevant log output

No response

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None
python: 3.8.5 (default, Sep 4 2020, 02:22:02)
[Clang 10.0.0 ]
python-bits: 64
OS: Darwin
OS-release: 20.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.10.4
libnetcdf: 4.7.3

xarray: 2022.3.0
pandas: 1.4.1
numpy: 1.22.3
scipy: 1.6.2
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.11.1
cftime: 1.5.1.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2021.10.0
distributed: 2021.10.0
matplotlib: 3.1.3
cartopy: None
seaborn: None
numbagg: None
fsspec: 2022.02.0
cupy: None
pint: None
sparse: None
setuptools: 60.7.1
pip: 22.0.3
conda: 4.11.0
pytest: 7.1.1
IPython: 7.31.1
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the Dataset.to_zarr entry point and read its compute=False documentation, especially the claim about returning a Dask.Delayed. Reproduce the minimal example with and without Dask, then trace how the returned object is executed. Done means the supported behavior for obtaining an awaitable in an async routine is clearly defined and covered by relevant tests or documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.