dask / dask/distributed

Serialize data within tasks

Open
#2,110 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently if we submit data within a task like the following:

```python
x = np.array([1, 2, 3])
future = client.submit(func, x)
```

Then we construct a task like `(func, x)` and then call pickle on this task. *We don't do custom serialization once we contruct tasks*. This is mostly because stuff in tasks is rarely large, and traversing tasks can increase our overhead in the common case. Typically we encourage people to use dask.delayed or something similar to mark data

```python
x = np.array([1, 2, 3])
x = dask.delayed(x)
future = client.submit(func, x)
```

But this is error prone and requires rarely held expertise.

We might again consider traversing arguments.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.