`xr.save_mfdataset()` doesn't honor `compute=False` argument
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened:
While using xr.save_mfdataset() function with compute=False I noticed that the function returns a dask.delayed object, but it doesn't actually defer the computation i.e. it actually writes datasets right away.
What you expected to happen:
I expect the datasets to be written when I explicitly call .compute() on the returned delayed object.
Minimal Complete Verifiable Example:
In [2]: import xarray as xr
In [3]: ds = xr.tutorial.open_dataset('rasm', chunks={})
In [4]: ds
Out[4]:
<xarray.Dataset>
Dimensions: (time: 36, x: 275, y: 205)
Coordinates:
* time (time) object 1980-09-16 12:00:00 ... 1983-08-17 00:00:00
xc (y, x) float64 dask.array<chunksize=(205, 275), meta=np.ndarray>
yc (y, x) float64 dask.array<chunksize=(205, 275), meta=np.ndarray>
Dimensions without coordinates: x, y
Data variables:
Tair (time, y, x) float64 dask.array<chunksize=(36, 205, 275), meta=np.ndarray>
Attributes:
title: /workspace/jhamman/processed/R1002RBRxaaa01a/l...
institution: U.W.
source: RACM R1002RBRxaaa01a
output_frequency: daily
output_mode: averaged
convention: CF-1.4
references: Based on the initial model of Liang et al., 19...
comment: Output from the Variable Infiltration Capacity...
nco_openmp_thread_number: 1
NCO: "4.6.0"
history: Tue Dec 27 14:15:22 2016: ncatted -a dimension...
In [5]: path = "test.nc"
In [7]: ls -ltrh test.nc
ls: cannot access test.nc: No such file or directory
In [8]: tasks = xr.save_mfdataset(datasets=[ds], paths=[path], compute=False)
In [9]: tasks
Out[9]: Delayed('list-aa0b52e0-e909-4e65-849f-74526d137542')
In [10]: ls -ltrh test.nc
-rw-r--r-- 1 abanihi ncar 14K Jul 8 10:29 test.nc
Anything else we need to know?:
Environment:
Output of xr.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.6 | packaged by conda-forge | (default, Jun 1 2020, 18:57:50)
[GCC 7.5.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-693.21.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.7.4
xarray: 0.15.1
pandas: 0.25.3
numpy: 1.18.5
scipy: 1.5.0
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: 2.10.0
Nio: None
zarr: None
cftime: 1.2.0
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.20.0
distributed: 2.20.0
matplotlib: 3.2.1
cartopy: None
seaborn: None
numbagg: None
setuptools: 49.1.0.post20200704
pip: 20.1.1
conda: None
pytest: None
IPython: 7.16.1
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the xr.save_mfdataset() entry point and reproduce the provided example with compute=False. Verify that no output file exists before the returned delayed object's .compute() is called, and that the datasets are written afterward; add a regression test for this behavior if the surrounding tests identify a suitable location.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100