Eagerly computing results limits potential for optimization
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
**Problem**
By eagerly computing, e.g., when calling `.head()`, we limit the potential for optimizations. For example, it shouldn't matter whether I call `df.head(n)[["a", "b"]]` or `df[["a", "b"]].head(n)`. With eager computations, we lose the opportunity to push the column projection before the head selection.
Apart from that, in my opinion that eager computation comes as a surprise and makes it harder to argue about when Dask actually computes things.
**Proposed solution(s)**
In general, limit eager computation as much as possible. For this particular example: Deprecate `compute=True` as default and switch to `compute=False` in the future.
Contributor guide
Research direction
No files or tests are named. Start by tracing the `.head()` example and the `compute=True`/`compute=False` behavior described in the issue. Done means eager computation is reduced as proposed, projection can be pushed before head selection, and the default change is handled through the stated deprecation path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100