dask / dask/distributed

Custom Serialization for Task Args

Open
#2,953 4 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

The arguments of a task submitted to the scheduler are currently serialized using `pickle` and will not use any custom serialization ( `warn_dumps ⟶ pickle.dumps` https://github.com/dask/distributed/issues/2110#issuecomment-405069639). This is also demonstrated by the below example.

```python
class Foo:
"""Some class which **cannot** be pickled"""
def __init__(self, bar):
self.bar = bar

def __setstate__(self, state):
raise ValueError('Seriously, I cannot be pickled!')

@dask_serialize.register(Foo)
def special_serializer(x, *args, **kwargs):
# ... magic way of serializing Foo into List[bytes]
return {'serializer': 'special_serde'}, serialized_foo

@dask_deserialize.register(Foo)
def special_deserializer(header, frames):
# ... magic way of deserializing into Foo
return deserialized_foo

register_serialization_family('special_serde', special_serializer, special_deserializer)
client = Client(serializers=['dask', 'special_serde'], deserializers=['dask', 'special_serde'], processes=False)

@delayed
def some_func(_foo):
return 1 + 1

val = some_func(Foo(2))
val.compute()
```

Will always raise the `ValueError` set in `Foo`.
_Originally posted by @milesgranger in https://github.com/dask/distributed/issues/2469#issuecomment-457245041_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.