dask / dask/distributed

Worker plugin can not be registered on worker unless its entire package source uploaded on server

Open
#8,695 7 comments 1 reaction 0 assignees View on GitHub
needs triage
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Describe the issue**:
Let's say I've created a plugin under `my_package.dask.plugins.MyWorkerPlugin`.

the plugin registration:
```
from my_package.dask.plugins import MyWorkerPlugin
client.register_plugin(MyWorkerPlugin())
```
will lead to:
```
File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 970, in _handle_comm
File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 7893, in register_worker_plugin
File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 6488, in broadcast
File "/opt/conda/lib/python3.10/site-packages/distributed/utils.py", line 251, in All
File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 6466, in send_message
File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1181, in send_recv
Exception: ModuleNotFoundError("No module named 'my_package'")
```

It would be useful if Dask could provide more meaningful error messages or upload the package automatically. However, the serialisation method should allow the use of the registered method.

**Environment**:

- Dask version: 2024.5.2
- Python version: 3.10
- Operating System: WSL
- Install method (conda, pip, source): poetry

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.