astronomy-commons / astronomy-commons/lsdb
Specify memory required to merge computed worker results
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
When creating a Dask Client we specify the number of workers and the memory limit for each of them. This means that each worker is assigned the same amount of memory. E.g.:
```
dask.distributed.Client(n_workers=16, memory_limit="4GiB")
```
When we hit `compute` one of those workers is responsible for merging the results of all individual computations and, if the result is bigger than that single worker's memory, the computation fails. We should be able to specify a different amount of memory required for a final worker to complete the computation.
There seems to be no way of specifying which merge worker to use on compute of a Dask DataFrame but that seems to be possible through the Client: https://distributed.dask.org/en/stable/api.html#distributed.Client.persist. Requires some investigation.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by investigating how Dask DataFrame compute chooses the worker that merges results, then compare that path with the distributed.Client and Client.persist APIs linked in the issue. Define how a caller would specify extra memory for the final worker and verify the behavior with a computation whose merged result exceeds the standard per-worker limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100