dask / dask/dask-expr

Smarter default for `split_out` in groupby aggregations

Open
#867 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

Currently, the default for `split_out` is based on the number of partitions and groupby columns:

https://github.com/dask-contrib/dask-expr/blob/fdeee4375df55499462bc35af50d3d216d5e256c/dask_expr/_groupby.py#L81-L82

Ideally, this would take (an estimate of) the number of unique groups into account to avoid situations like https://github.com/coiled/benchmarks/issues/1376.

Contributor guide

Open the contributing guide

Research direction

Start by reading dask_expr/_groupby.py at the linked lines to understand the current split_out calculation. Review the linked benchmark issue for the motivating situation and investigate how unique-group counts could be estimated. Done means the default accounts for that estimate while preserving correct groupby aggregation behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.