dask / dask/dask-expr

Eagerly computing results limits potential for optimization

Open
#826 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

**Problem**

By eagerly computing, e.g., when calling `.head()`, we limit the potential for optimizations. For example, it shouldn't matter whether I call `df.head(n)[["a", "b"]]` or `df[["a", "b"]].head(n)`. With eager computations, we lose the opportunity to push the column projection before the head selection.

Apart from that, in my opinion that eager computation comes as a surprise and makes it harder to argue about when Dask actually computes things.

**Proposed solution(s)**

In general, limit eager computation as much as possible. For this particular example: Deprecate `compute=True` as default and switch to `compute=False` in the future.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing the `.head()` example and the `compute=True`/`compute=False` behavior described in the issue. Done means eager computation is reduced as proposed, projection can be pushed before head selection, and the default change is handled through the stated deprecation path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.