Analyse shouldn't repartition to a single partition by default
Open
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
analyse currently repartitions to a single partition, which will blow your computation up if you are running this on a query that isn't reducing very heavily, see
https://github.com/dask/dask/issues/11307
This shouldn't be the default behavior
cc @hendrikmakait
Contributor guide
Research direction
Start by tracing the analyse entry point and how it currently invokes repartition, then reproduce the failure mode described in the linked Dask issue with a query that does not reduce heavily. Done means analyse no longer repartitions to a single partition by default while preserving its intended analysis behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100