dask / dask/distributed

RFC: explicit shared memory

Open
#4,497 24 comments 6 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

With the increasing availability of large machines, it seems to be the case that more workloads are being run as many processes on a single node. In a workflow where a single large array would be passed to workers, currently this might be done by passing the array from the client (bad), using `scatter` (OK) or loading data in the workers (good, but not efficient if we want one big array).

A large memory and transfer cost might be saved by putting the array into posix shared memory and referencing it from the workers. If we host the array is in shm, there is no copy or de/ser cost (but there is an OS call cost to attach to the shm). It could be appropriate for ML workflows where every task wants to make use of the whole of a large dataset (as opposed to chunking the dataset as dask.array operations do). sklearn with joblib is an example where we explicitly recommend scattering large data.

As a really simple example, see [my gist](https://gist.github.com/martindurant/5f517ec55a5bff9c32637e8ebc57ef7c), in which the user has to explicitly wrap a numpy array in the client, and then dask workers no longer need to have their own copies. Note that `SharedArray` is just a simple way to pass the array metadata as well as its buffer; it works for py37 and probably earlier.

To be clear: there is no suggestion of adding anything to the existing distributed serialisation code, because it's *really hard* to try to guess when a user might want to use such a thing. It should be explicitly opt-in.

Further,
- Similar techniques could be used to wrap arrow or pandas data, although no one probably wants to delve through existing in-memory objects to find the underlying buffers.
- Pickle V5 works on buffers and memoryviews, so might be a generic helper here

cc @crusaderky @quasiben @jsignell

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.