dask / dask/distributed

Serialize `numba.typed.List`

Open
#5,781 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:

I am trying to call a `numba` `njit` function that requires a `numba.typed.List` as a parameter. Eventually, this `numba.typed.List` will be filled with `numpy` arrays of variable length (think jagged array and hence why we are using a list). Calling this `numba` function works without `dask` but it fails when `dask` is used and I am getting a:

```
TypeError: ('Could not serialize object of type List.', '[[], [], [], [], [], [], [], [], [], []]')
```

**What you expected to happen**:

I (naively) expect to be able to serialize this `numba.typed.List` and then be able to return the `numba.typed.List`

**Minimal Complete Verifiable Example**:

```python
import numpy as np
from dask.distributed import Client, LocalCluster
from numba.typed import List
from numba import njit

@njit()
def _some_func(T, A):
# Do something with T and A
# Maybe fill A with something like
# for i in range(len(T)):
# A[i] = T[ : i]
return A

if __name__ == "__main__":
dask_cluster = LocalCluster(n_workers=2, threads_per_worker=2)
T = np.random.rand(10)
n = T.shape[0]
A = List([np.empty(0, dtype=np.float64) for _ in range(n)])

with Client(dask_cluster) as dask_client:
T_future = dask_client.scatter(T, broadcast=True, hash=False)
A_future = dask_client.scatter(A, broadcast=True, hash=False)

future = dask_client.submit(
_some_func,
T_future,
A_future,
)

results = dask_client.gather(future)
dask_client.cancel(T_future)
dask_client.cancel(A_future)
```

**Anything else we need to know?**:

**Environment**:

- Dask version:
```
dask 2022.1.1 pyhd8ed1ab_0 conda-forge
dask-core 2022.1.1 pyhd8ed1ab_0 conda-forge
distributed 2022.1.1 py39h2804cbe_0 conda-forge
cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge
```
- Python version: `3.9.10`
- Operating System: `Mac OS`
- Install method (conda, pip, source): `conda`

Cluster Dump State:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.