Smarter default for `split_out` in groupby aggregations
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
Currently, the default for `split_out` is based on the number of partitions and groupby columns:
https://github.com/dask-contrib/dask-expr/blob/fdeee4375df55499462bc35af50d3d216d5e256c/dask_expr/_groupby.py#L81-L82
Ideally, this would take (an estimate of) the number of unique groups into account to avoid situations like https://github.com/coiled/benchmarks/issues/1376.
Contributor guide
Research direction
Start by reading dask_expr/_groupby.py at the linked lines to understand the current split_out calculation. Review the linked benchmark issue for the motivating situation and investigate how unique-group counts could be estimated. Done means the default accounts for that estimate while preserving correct groupby aggregation behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100