dask / dask/distributed

Alternative destination-specific serialization

Open
#400 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

In some cases objects may want to serialize themselves dependent on the sender and recipient machine. One such example is multi-gpu settings where certain workers may be able to exchange GPU data efficiently if they are on the same physical host or nearby in the network.

To meet this goal it may make sense to enable host-specific serialization, that is we would optionally use a new serialization function that was invoked for certain types:

```
serialize(obj, host, destination) -> bytes
```

We would probably have to register serialization functions for certain types or else look for a special protocol like `__getstate_host__` and `__setstate_host__`.

cc @seibert @sklam @gforsyth

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.