Preprocess function for save_mfdataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem? Please describe.
I would like to supply a preprocess argument to save_mfdataset that gets applied to each dataset before getting written to disk, similar to how open_mfdataset gives you such option. Specifically, have a dataset that I want to split by unique values along dimension, apply some further logic to each sub-dataset, then save each sub-dataset to a different file. Currently I'm able to split and save using the following code provided in the API docs:
years, datasets = zip(*ds.groupby("time.year"))
paths = ["%s.nc" % y for y in years]
xr.save_mfdataset(datasets, paths)
What's missing is the ability to insert further logic to each of the sub-datasets given by the groupby object. If I try iterating through datasets here and chain further operations to each element, the calculations begin to execute serially even though ds is a dask array:
save_mfdataset([ds.foo() for ds in datasets], paths)
Describe the solution you'd like
Instead, I'd like the ability to do:
xr.save_mfdataset(datasets, paths, preprocess=lambda ds: ds.foo())
Describe alternatives you've considered
Not sure.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the save_mfdataset and open_mfdataset entry points to understand how preprocessing is exposed, then trace the save_mfdataset path used by the documented groupby example. Add support for the requested preprocess argument and verify that it is applied to each dataset without forcing the dask-backed calculations to execute serially.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100