astronomy-commons / astronomy-commons/lsdb

Specify memory required to merge computed worker results

Open
#678 1 comment 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

When creating a Dask Client we specify the number of workers and the memory limit for each of them. This means that each worker is assigned the same amount of memory. E.g.:

```
dask.distributed.Client(n_workers=16, memory_limit="4GiB")
```

When we hit `compute` one of those workers is responsible for merging the results of all individual computations and, if the result is bigger than that single worker's memory, the computation fails. We should be able to specify a different amount of memory required for a final worker to complete the computation.

There seems to be no way of specifying which merge worker to use on compute of a Dask DataFrame but that seems to be possible through the Client: https://distributed.dask.org/en/stable/api.html#distributed.Client.persist. Requires some investigation.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by investigating how Dask DataFrame compute chooses the worker that merges results, then compare that path with the distributed.Client and Client.persist APIs linked in the issue. Define how a caller would specify extra memory for the final worker and verify the behavior with a computation whose merged result exceeds the standard per-worker limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.