dask / dask/distributed

Better error for missing modules during deserialization

Open
#5,419 3 comments 1 reaction 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Beginning users are often tripped up by environment management in remote clusters. Trying to run a function from a module which doesn't exist can throw a long and cryptic `ModuleNotFoundError` or `ImportError`, which users will have a hard time interpreting (as discussed in https://github.com/dask/distributed/issues/4880).

For example, on the [Coiled community slack](https://coiled-users.slack.com/archives/C0195GJKQ1G/p1634051823141900), a user reported:

> I've been trying today to run encoding in a cluster and I've been getting a strange error (I think not coiled related) from the workers:
> ```
> [ 734.614436] cloud-init[1027]: distributed.protocol.core - CRITICAL - Failed to deserialize
> [ 734.614485] cloud-init[1027]: Traceback (most recent call last):
> [ 734.614525] cloud-init[1027]: File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
> [ 734.614573] cloud-init[1027]: return msgpack.loads(
> [ 734.614619] cloud-init[1027]: File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
> [ 734.614661] cloud-init[1027]: File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
> [ 734.614700] cloud-init[1027]: return merge_and_deserialize(
> [ 734.614740] cloud-init[1027]: File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
> [ 734.614780] cloud-init[1027]: return deserialize(header, merged_frames, deserializers=deserializers)
> [ 734.614820] cloud-init[1027]: File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
> [ 734.614859] cloud-init[1027]: deserialize(
> [ 734.614902] cloud-init[1027]: File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
> [ 734.614941] cloud-init[1027]: return loads(header, frames)
> [ 734.614980] cloud-init[1027]: File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
> [ 734.615020] cloud-init[1027]: return pickle.loads(x, buffers=new)
> [ 734.615059] cloud-init[1027]: File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 75, in loads
> [ 734.615095] cloud-init[1027]: return pickle.loads(x)
> [ 734.615132] cloud-init[1027]: ModuleNotFoundError: No module named 'encode'
> ```
> Something with pickle and encoding, I'm wondering if you have ever encountered this? Most likely this is something messed up in my environment but wanted to check with you.

It took many back-and-forth messages to discover that the user had a separate `encode.py` file where their `@delayed` functions were defined.

What if instead the error had looked like:

```
ImportError: The package named 'encode' is not installed on this cluster.

If 'encode' is a package installed from pip/conda:

You can install 'encode' in your Coiled software environment by adding it to the list of pip
or conda packages and recreating it, like:
>>> coiled.create_software_environment(name="...", pip=[..., "encode"])
See https://docs.coiled.io/user_guide/software_environment_creation.html for more information.

Or, you can install it temporarily on this cluster using a `PipInstall` plugin [^1]:
>>> from dask.distributed import PipInstall
>>> plugin = PipInstall(packages=["encode"], pip_options=["--upgrade"])
>>> client.register_worker_plugin(plugin)
Note that you will have to rerun this every time you recreate the cluster.

If 'encode.py' is file on your local machine:

To temporarily upload this file to the cluster, use `client.upload_file` [^2]:
>>> client.upload_file('encode.py')

If you need multiple files, or 'encode.py' is part of a local package,
use the `UploadDirectoryPlugin` [^3]:
>>> from distributed.diagnostics.plugin import UploadDirectory
>>> client.register_worker_plugin(UploadDirectory("/path/to/directory"), nanny=True)

Note that you will have to rerun either of these every time you recreate the cluster.

In general, you must have the same imports available on your cluster as on your local machine. See
https://docs.dask.org/en/stable/setup/environment.html for more information on this.

[^1]: https://distributed.dask.org/en/stable/plugins.html#distributed.diagnostics.plugin.PipInstall
[^2]: https://distributed.dask.org/en/stable/api.html#distributed.Client.upload_file
[^3]: https://distributed.dask.org/en/stable/plugins.html#distributed.diagnostics.plugin.UploadDirectory
```

Notice my message includes a "You can install 'encode' in your Coiled software environment..." section. Obviously this wouldn't be hardcoded into dask. Instead, I'm imagining a `distributed.errors.import_error_template` config variable, which would contain a format-string template which different cluster managers (dask-gateway, Pangeo, Coiled, Saturn, etc.) could customize via an environment variable or `dask.yaml` file, to include custom instructions about installing dependencies within that system.

xref https://github.com/dask/distributed/issues/2230

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.