Worker plugin can not be registered on worker unless its entire package source uploaded on server
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**Describe the issue**:
Let's say I've created a plugin under `my_package.dask.plugins.MyWorkerPlugin`.
the plugin registration:
```
from my_package.dask.plugins import MyWorkerPlugin
client.register_plugin(MyWorkerPlugin())
```
will lead to:
```
File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 970, in _handle_comm
File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 7893, in register_worker_plugin
File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 6488, in broadcast
File "/opt/conda/lib/python3.10/site-packages/distributed/utils.py", line 251, in All
File "/opt/conda/lib/python3.10/site-packages/distributed/scheduler.py", line 6466, in send_message
File "/opt/conda/lib/python3.10/site-packages/distributed/core.py", line 1181, in send_recv
Exception: ModuleNotFoundError("No module named 'my_package'")
```
It would be useful if Dask could provide more meaningful error messages or upload the package automatically. However, the serialisation method should allow the use of the registered method.
**Environment**:
- Dask version: 2024.5.2
- Python version: 3.10
- Operating System: WSL
- Install method (conda, pip, source): poetry
Contributor guide
Assessment
This issue has not been assessed yet.