dask / dask/dask-cloudprovider
ECSCluster does not de-provision tasks after failing to connect to scheduler
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
**What happened**:
If ECSCluster successfully runs a scheduler task but then fails to connect to the scheduler, an error is raised and the scheduler task is left active. The task has to be manually removed from the ECS cluster and deregistered.
In my case, this happened when I forgot to connect to a VPN network that would provide connectivity to the VPC. The task successfully ran, but no network connection to the scheduler was possible.
```
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/distributed/comm/core.py", line 286, in connect
comm = await asyncio.wait_for(
File "/usr/local/lib/python3.8/asyncio/tasks.py", line 498, in wait_for
raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.8/site-packages/dask_cloudprovider/aws/ecs.py", line 729, in __init__
super().__init__(**kwargs)
File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 281, in __init__
self.sync(self._start)
File "/usr/local/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 189, in sync
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 340, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 324, in f
result[0] = yield future
File "/usr/local/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.8/site-packages/dask_cloudprovider/aws/ecs.py", line 926, in _start
await super()._start()
File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 314, in _start
await super()._start()
File "/usr/local/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 73, in _start
comm = await self.scheduler_comm.live_comm()
File "/usr/local/lib/python3.8/site-packages/distributed/core.py", line 746, in live_comm
comm = await connect(
File "/usr/local/lib/python3.8/site-packages/distributed/comm/core.py", line 308, in connect
raise IOError(
OSError: Timed out trying to connect to tcp://10.53.13.110:8786 after 10 s
```
**What you expected to happen**:
If ECSCluster fails to connect to the running scheduler, it should catch the error and clean up the scheduler task.
**Minimal Complete Verifiable Example**:
Create an `ECSCluster` instance with tasks in an unreachable network.
**Anything else we need to know?**:
None.
**Environment**:
- Dask version: 2012.2.0
- Python version: 3.8.6
- Operating System: Debian (python:3.8 Docker image)
- Install method (conda, pip, source): pip
Contributor guide
Assessment
This issue has not been assessed yet.