dask / dask/dask-cloudprovider
dask_cloudprovider.utils.timeout.TimeoutException: Failed to find scheduler ip address after 120 seconds
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
I am getting timeout after launching Fargate cluster
`dask_cloudprovider.utils.timeout.TimeoutException: Failed to find scheduler ip address after 120 seconds.`
I know this error has been reported before, but in my case it doesn't seem to be related to timeout duration as I get the same error after waiting for 5min
Looking into the source code it seems that the issue is due to parsing and getting IP from logs at https://github.com/dask/dask-cloudprovider/blob/main/dask_cloudprovider/aws/ecs.py#L209
It tries to access events within logs but it is empty when I print it at this line https://github.com/dask/dask-cloudprovider/blob/main/dask_cloudprovider/aws/ecs.py#L379
```
{'events': [], 'nextForwardToken': 'f/36723491871038569079854395880864976970354492776673968128/s', 'nextBackwardToken': 'b/36723331305673139659367776823693413902151649717678768128/s', 'ResponseMetadata': {'RequestId': 'a4af94b5-b125-44c5-8b22-8455cf0c07e4', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'a4af94b5-b125-44c5-8b22-8455cf0c07e4', 'content-type': 'application/x-amz-json-1.1', 'content-length': '178', 'date': 'Tue, 08 Mar 2022 09:18:47 GMT'}, 'RetryAttempts': 0}}
```
**What happened**:
TimeoutException
**What you expected to happen**:
No failure
**Minimal Complete Verifiable Example**:
```python
cluster = FargateCluster(
image="prefecthq/prefect:latest",
execution_role_arn="arn:aws:iam::**:role/dusk-cluster",
task_role_policies = ["arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"],
platform_version='1.3.0',
n_workers=3,
scheduler_cpu=256,
scheduler_mem=512,
worker_cpu=256,
worker_mem=512,
scheduler_timeout="15 minutes",
find_address_timeout=120
)
```
**Anything else we need to know?**:
**Environment**:
- Dask version: dask-cloudprovider-2022.1.0, dask-2022.2.1
- Python version: Python 3.8.12
- Operating System: Ubuntu 18.04.6 LTS
- Install method (conda, pip, source): Poetry
Cluster Dump State:
Contributor guide
Assessment
This issue has not been assessed yet.