dask / dask/distributed

Expunge unresponsive workers in less than 10s

Open
#5,585 1 comment 0 reactions 0 assignees View on GitHub
bug p3
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I'm writing a test that waits for the scheduler to remove an unresponsive worker.
I can't seem to drop its runtime below 9.8 seconds:

```python
@gen_cluster(
client=True,
Worker=Nanny,
nthreads=[("", 1)],
config={"distributed.scheduler.worker-ttl": "500ms"}, # This is largely inconsequential
)
async def test1(c, s, nanny):
addr, = s.workers
stuck_fut = asyncio.create_task(c.run(sleep, 3600, workers=[addr]))
while addr in s.workers:
await asyncio.sleep(0.01)
```

End-to-end runtime, as measured by pytest: 9.8s
Breakdown:
- 0.9s gen_cluster setup - time to reach first line of the test
- 4.0s time to reach log line "WARNING - Worker failed to heartbeat within 0.5 seconds. Closing"
- 0.0s time to reach past last line of the test
- 4.9s gen_cluster teardown

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.