dask / dask/distributed

FargateCluster timeout on exit

Open
#5,447 8 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

When I close a Fargate cluster (similar to [#220](https://github.com/dask/dask-cloudprovider/issues/220#issue-765269946) )
using

```python
client.close()
cluster.close()
```
I receive the following error
```
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/site-packages/distributed/deploy/adaptive_core.py", line 190, in adapt
target = await self.safe_target()
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/site-packages/distributed/deploy/adaptive_core.py", line 128, in safe_target
n = await self.target()
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/site-packages/distributed/deploy/adaptive.py", line 146, in target
return await self.scheduler.adaptive_target(
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/site-packages/distributed/core.py", line 789, in send_recv_from_rpc
comm = await self.live_comm()
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/site-packages/distributed/core.py", line 747, in live_comm
comm = await connect(
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/site-packages/distributed/comm/core.py", line 307, in connect
raise IOError(
OSError: Timed out trying to connect to tcp://3.87.54.191:8786 after 10 s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
ret = callback()
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
future.result()
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/site-packages/distributed/deploy/adaptive_core.py", line 204, in adapt
if status != "down":
UnboundLocalError: local variable 'status' referenced before assignment
tornado.application - ERROR - Exception in callback functools.partial(>, exception=UnboundLocalError("local variable 'status' referenced before assignment")>)
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/site-packages/distributed/comm/core.py", line 285, in connect
comm = await asyncio.wait_for(
File "/home/ec2-user/anaconda3/envs/features_r/lib/python3.8/asyncio/tasks.py", line 490, in wait_for
raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.