dask / dask/dask

querying df.compute(concatenate=True)

Open
#11,768 8 comments 3 reactions 0 assignees View on GitHub
needs attention needs triage
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

https://github.com/dask/dask-expr/pull/1138 introduced the `concatenate` kwargs to dask-dataframe compute operations, and defaulted to True (a change in behaviour). This is now the default in core dask following the merger of expr into the main repo.

I am concerned that the linked PR did not provide any rationale for the change, nor document under what circumstances it should *not* be used.

> Concatenating enables more powerful optimizations but it also incurs additional
> data transfer cost. Generally, it should be enabled.

I suggest the following contraindications:
- worker memory limits are generally much more strict than in the client, so concatenating in-cluster can crash the specific worker and make the workflow unrunnable
- the concatenation task cannot begin until all of its inputs are ready, whereas the client can download each partition as it completes, so in the straggler case, concatenate=True will tend to be slower

I can see the option being useful in the case that:
- there are a large number of small partitions in the output, and we expect the inter-worker latency to be much more favourable than the client-worker latency

I can see the option making no difference in the case that:
- the number of partitions is small compared to the total volume of data in the output, but there is no worker memory issue

cf https://github.com/dask/community/issues/411

Contributor guide

Open the contributing guide

Research direction

Start with the dask-dataframe compute operations and the dask-expr pull request #1138, then compare the current core Dask behavior. Document when concatenate=True helps, its worker-memory and straggler tradeoffs, and when it may be unnecessary; use community issue #411 for related context.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, distributed-systems
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.