WorkerPlugin not re-registered after scheduler reconnected
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
Dask distributed client is capable of re-connecting to the scheduler if the connection is lost. So I decided to restart the scheduler every time scheduler hangs (discussed later). However, I found my WorkerPlugin registered from client was not re-registered to workers later connected to scheduler. To reproduce in simplest steps:
1. Start scheduler
2. Start client, which registers a WorkerPlugin
3. Kill scheduler. Now client.status changes from 'running' to 'connecting'
5. Before client gives up reconnecting, re-start scheduler
6. Connect a worker
7. WorkerPlugin not registered.
**What you expected to happen**:
Because the client reconnects to scheduler automatically, to the user/application it is the same client. Therefore the user should expect that the WorkerPlugin is re-registered.
**Minimal Complete Verifiable Example**:
Here I extract essential parts of my codes:
```python
import dask
import dask.distributed as dd
import sys
class MyPlugin(dd.WorkerPlugin):
def setup(self, worker: dd.Worker):
print('WorkerPlugin setup!', file=sys.__stdio__)
master='localhost:8786'
dask_client = dd.Client(master, direct_to_workers=True)
dask_client.register_worker_plugin(MyPlugin(), name='myplugin')
...
```
Use above steps to reproduce.
**Why restart the scheduler?**
As scheduler runs and handles tasks over time, the data retrieval time starts to slow down. (Tasks were done and in memory but it takes long time to get result back.) Note, my results are within KBs, and if I restart the scheduler, all comes back to normal. I guess it is due to many congested tasks in scheduler's asyncio framework. I tried direct_to_worker but it seemed not helpful. Like many similar reports, this is hard to reproduce in a simple way either.
**Environment**:
- Dask version: 2021.7.0
- Python version: 3.7.0
- Operating System: Windows 10
- Install method (conda, pip, source): conda install -c conda-forge dask==2021.7.0
Contributor guide
Assessment
This issue has not been assessed yet.