Expunge unresponsive workers in less than 10s
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I'm writing a test that waits for the scheduler to remove an unresponsive worker.
I can't seem to drop its runtime below 9.8 seconds:
```python
@gen_cluster(
client=True,
Worker=Nanny,
nthreads=[("", 1)],
config={"distributed.scheduler.worker-ttl": "500ms"}, # This is largely inconsequential
)
async def test1(c, s, nanny):
addr, = s.workers
stuck_fut = asyncio.create_task(c.run(sleep, 3600, workers=[addr]))
while addr in s.workers:
await asyncio.sleep(0.01)
```
End-to-end runtime, as measured by pytest: 9.8s
Breakdown:
- 0.9s gen_cluster setup - time to reach first line of the test
- 4.0s time to reach log line "WARNING - Worker failed to heartbeat within 0.5 seconds. Closing"
- 0.0s time to reach past last line of the test
- 4.9s gen_cluster teardown
Contributor guide
Assessment
This issue has not been assessed yet.