Custom Serialization for Task Args
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
The arguments of a task submitted to the scheduler are currently serialized using `pickle` and will not use any custom serialization ( `warn_dumps ⟶ pickle.dumps` https://github.com/dask/distributed/issues/2110#issuecomment-405069639). This is also demonstrated by the below example.
```python
class Foo:
"""Some class which **cannot** be pickled"""
def __init__(self, bar):
self.bar = bar
def __setstate__(self, state):
raise ValueError('Seriously, I cannot be pickled!')
@dask_serialize.register(Foo)
def special_serializer(x, *args, **kwargs):
# ... magic way of serializing Foo into List[bytes]
return {'serializer': 'special_serde'}, serialized_foo
@dask_deserialize.register(Foo)
def special_deserializer(header, frames):
# ... magic way of deserializing into Foo
return deserialized_foo
register_serialization_family('special_serde', special_serializer, special_deserializer)
client = Client(serializers=['dask', 'special_serde'], deserializers=['dask', 'special_serde'], processes=False)
@delayed
def some_func(_foo):
return 1 + 1
val = some_func(Foo(2))
val.compute()
```
Will always raise the `ValueError` set in `Foo`.
_Originally posted by @milesgranger in https://github.com/dask/distributed/issues/2469#issuecomment-457245041_
Contributor guide
Assessment
This issue has not been assessed yet.