dask / dask/distributed

Pickling error with numba gufunc

Open
#7,929 6 comments 0 reactions 2 assignees Claimed by @crusaderky View on GitHub
bug
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I'm trying [this example from the dask-examples repo](https://github.com/dask/dask-examples/blob/main/applications/stencils-with-numba.ipynb). When computing the graph it gives a `PicklingError`
```
_pickle.PicklingError: Can't pickle : it's not the same object as __main__.smooth
```
This happens only when using distributed. I launch a cluster in a Slurm cluster via `ipyparallel`'s `become_dask`. Something like [this](https://ipyparallel.readthedocs.io/en/latest/examples/dask.html). Without using distributed the example works fine.

These are the versions of some relevant packages I'm using:
```
dask-2023.6.0
distributed-2023.6.0
toolz-0.12.0
cloudpickle-2.2.1
msgpack-1.0.5
numba-0.57.0
```

Not sure if this is a bug. Probably it's something in my environment. I found similar serialization errors but nothing recent. Only solved issues earlier than 2022.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.