Serialize `numba.typed.List`
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
I am trying to call a `numba` `njit` function that requires a `numba.typed.List` as a parameter. Eventually, this `numba.typed.List` will be filled with `numpy` arrays of variable length (think jagged array and hence why we are using a list). Calling this `numba` function works without `dask` but it fails when `dask` is used and I am getting a:
```
TypeError: ('Could not serialize object of type List.', '[[], [], [], [], [], [], [], [], [], []]')
```
**What you expected to happen**:
I (naively) expect to be able to serialize this `numba.typed.List` and then be able to return the `numba.typed.List`
**Minimal Complete Verifiable Example**:
```python
import numpy as np
from dask.distributed import Client, LocalCluster
from numba.typed import List
from numba import njit
@njit()
def _some_func(T, A):
# Do something with T and A
# Maybe fill A with something like
# for i in range(len(T)):
# A[i] = T[ : i]
return A
if __name__ == "__main__":
dask_cluster = LocalCluster(n_workers=2, threads_per_worker=2)
T = np.random.rand(10)
n = T.shape[0]
A = List([np.empty(0, dtype=np.float64) for _ in range(n)])
with Client(dask_cluster) as dask_client:
T_future = dask_client.scatter(T, broadcast=True, hash=False)
A_future = dask_client.scatter(A, broadcast=True, hash=False)
future = dask_client.submit(
_some_func,
T_future,
A_future,
)
results = dask_client.gather(future)
dask_client.cancel(T_future)
dask_client.cancel(A_future)
```
**Anything else we need to know?**:
**Environment**:
- Dask version:
```
dask 2022.1.1 pyhd8ed1ab_0 conda-forge
dask-core 2022.1.1 pyhd8ed1ab_0 conda-forge
distributed 2022.1.1 py39h2804cbe_0 conda-forge
cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge
```
- Python version: `3.9.10`
- Operating System: `Mac OS`
- Install method (conda, pip, source): `conda`
Cluster Dump State:
Contributor guide
Assessment
This issue has not been assessed yet.