Using the shuffle primitive in Xarray
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem?
dask recently added dask.array.shuffle to help with some classic GroupBy.map problems.
shuffle reorders the array so that all members of a single group are in a single chunk, with the possibility of multiple groups in a single chunk. I see a few ways to use this in Xarray:
GroupBy.shuffle()This shuffles and returns a new GroupBy object with which to do further operations (e.g.map).Dataset.shuffle_by(Grouper)This shuffles, and returns a new dataset (or dataarray), so that the shuffled data can be persisted to disk or you can do other things later (xref #5674)- Use
GroupBy.shuffleunder the hood inDatasetGroupBy.quantileandDatasetGroupBy.median, so that the exact quantile always works regardless of chunking (right now we raise and error), this seems like a no-brainer. - Add either a
shufflekwarg toGroupBy.mapand/orGroupBy.reduceor a new API (e.g.GroupBy.transformorGroupBy.map_shuffled) that will shuffle, thenxarray.map_blocksa wrapper function that applies theGroupbyon each block. This is how dask dataframe implementsGroupby.apply
#9320 implements (1,2). (1) is mostly for convenience, I could easily see us recommending using (2) before calling the GroupBy.
Thoughts?
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the dask.array.shuffle documentation and the GroupBy.map and GroupBy.reduce APIs described in the issue. Review issue #9320 to understand the implemented shuffle approaches, then determine which remaining API or quantile/median behavior is intended; done requires a settled design and corresponding implementation direction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100