dask / dask/dask-expr

Analyse shouldn't repartition to a single partition by default

Open
#1,123 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

analyse currently repartitions to a single partition, which will blow your computation up if you are running this on a query that isn't reducing very heavily, see

https://github.com/dask/dask/issues/11307

This shouldn't be the default behavior

cc @hendrikmakait

Contributor guide

Open the contributing guide

Research direction

Start by tracing the analyse entry point and how it currently invokes repartition, then reproduce the failure mode described in the linked Dask issue with a query that does not reduce heavily. Done means analyse no longer repartitions to a single partition by default while preserving its intended analysis behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.