querying df.compute(concatenate=True)
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
https://github.com/dask/dask-expr/pull/1138 introduced the `concatenate` kwargs to dask-dataframe compute operations, and defaulted to True (a change in behaviour). This is now the default in core dask following the merger of expr into the main repo.
I am concerned that the linked PR did not provide any rationale for the change, nor document under what circumstances it should *not* be used.
> Concatenating enables more powerful optimizations but it also incurs additional
> data transfer cost. Generally, it should be enabled.
I suggest the following contraindications:
- worker memory limits are generally much more strict than in the client, so concatenating in-cluster can crash the specific worker and make the workflow unrunnable
- the concatenation task cannot begin until all of its inputs are ready, whereas the client can download each partition as it completes, so in the straggler case, concatenate=True will tend to be slower
I can see the option being useful in the case that:
- there are a large number of small partitions in the output, and we expect the inter-worker latency to be much more favourable than the client-worker latency
I can see the option making no difference in the case that:
- the number of partitions is small compared to the total volume of data in the output, but there is no worker memory issue
cf https://github.com/dask/community/issues/411
Contributor guide
Research direction
Start with the dask-dataframe compute operations and the dask-expr pull request #1138, then compare the current core Dask behavior. Document when concatenate=True helps, its worker-memory and straggler tradeoffs, and when it may be unnecessary; use community issue #411 for related context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, distributed-systems
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100