dask / dask/distributed

Handling serialization registration externally

Open
#3,831 10 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently custom serialization registration must be done [within Distributed itself]( https://github.com/dask/distributed/blob/master/distributed/protocol/__init__.py ). However this is a bit tricky to manage (particularly for newcomers).

First they need to implement serialization for their libraries objects [like so]( https://github.com/rapidsai/cudf/blob/cc414f97002768e7bec926b77a6ec1ba58c5fada/python/cudf/cudf/comm/serialize.py#L9-L28 ). Second they need to add code to distributed to [register their serializers]( https://github.com/dask/distributed/blob/43ad2f8cfaa216588f15d39dc4ea24c58ac2c633/distributed/protocol/__init__.py#L104-L109 ). Third they need to test this somehow using a development install of their library and Distributed.

It would be useful if this registration (second) step could be done in external libraries. This would allow all of the changes in one place without requiring some coordinations of changes, PRs, and releases so as not to break anything.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.