Refactor `xr.save_mfdataset()` to automatically save an xarray object backed by dask arrays to multiple files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem? Please describe.
Currently, when a user wants to write multiple netCDF files in parallel with xarray and dask, they can take full advantage of xr.save_mfdataset() function. This function in its current state works fine, but the existing API requires that
- the user generates file paths themselves
- the user maps each chunk or dataset to a corresponding output file
A few months ago, I wrote a blog post showing how to save an xarray dataset backed by dask into multiple netCDF files, and since then I've been meaning to request a new feature to make this process convenient for users.
Describe the solution you'd like
Would it be useful to actually refactor the existing xr.save_mfdataset() to automatically save an xarray object backed by dask arrays to multiple files without needing to create paths ourselves? Today, this can be achieved via xr.map_blocks. In other words, is it possible to have something analogous to to_zarr(....) but for netCDF:
ds.save_mfdataset(prefix="directory/my-dataset")
# or
xr.save_mfdataset(ds, prefix="directoy/my-dataset")
---->
directory/my-dataset-chunk-1.nc
directory/my-dataset-chunk-2.nc
directory/my-dataset-chunk-3.nc
....
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing xr.save_mfdataset() API and the xr.map_blocks approach described in the issue. Clarify how chunk-to-file mapping and generated paths should work; done means an xarray object backed by dask arrays can be saved through the proposed prefix-based API into multiple netCDF files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100