dask / dask/distributed

WorkerPlugin not re-registered after scheduler reconnected

Open
#5,125 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
Dask distributed client is capable of re-connecting to the scheduler if the connection is lost. So I decided to restart the scheduler every time scheduler hangs (discussed later). However, I found my WorkerPlugin registered from client was not re-registered to workers later connected to scheduler. To reproduce in simplest steps:

1. Start scheduler
2. Start client, which registers a WorkerPlugin
3. Kill scheduler. Now client.status changes from 'running' to 'connecting'
5. Before client gives up reconnecting, re-start scheduler
6. Connect a worker
7. WorkerPlugin not registered.

**What you expected to happen**:
Because the client reconnects to scheduler automatically, to the user/application it is the same client. Therefore the user should expect that the WorkerPlugin is re-registered.

**Minimal Complete Verifiable Example**:
Here I extract essential parts of my codes:

```python
import dask
import dask.distributed as dd
import sys

class MyPlugin(dd.WorkerPlugin):
def setup(self, worker: dd.Worker):
print('WorkerPlugin setup!', file=sys.__stdio__)

master='localhost:8786'
dask_client = dd.Client(master, direct_to_workers=True)
dask_client.register_worker_plugin(MyPlugin(), name='myplugin')
...
```
Use above steps to reproduce.

**Why restart the scheduler?**
As scheduler runs and handles tasks over time, the data retrieval time starts to slow down. (Tasks were done and in memory but it takes long time to get result back.) Note, my results are within KBs, and if I restart the scheduler, all comes back to normal. I guess it is due to many congested tasks in scheduler's asyncio framework. I tried direct_to_worker but it seemed not helpful. Like many similar reports, this is hard to reproduce in a simple way either.

**Environment**:

- Dask version: 2021.7.0
- Python version: 3.7.0
- Operating System: Windows 10
- Install method (conda, pip, source): conda install -c conda-forge dask==2021.7.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.