dask / dask/dask-cloudprovider

FargateCluster defaults require too many VCPUs for a new AWS account

Open
#299 3 comments 0 reactions 0 assignees View on GitHub
documentation hygiene provider/aws/ecs
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

I created a new AWS account to experiment with dask, but I can't create a cluster with any workers without specifying CPU settings:

```python
from dask_cloudprovider.aws import FargateCluster
cluster = FargateCluster(n_workers=1)
```
Output:

```
/opt/anaconda3/lib/python3.8/contextlib.py:120: UserWarning: Creating your cluster is taking a surprisingly long time. This is likely due to pending resources on AWS. Hang tight!
next(self.gen)
Task exception was never retrieved
future: exception=RuntimeError({'tasks': [], 'failures': [{'reason': **'The requested CPU configuration is above your limit'**}], 'ResponseMetadata': {'RequestId': '3e739dbb-4c9c-4d4b-b37d-a02ef76897c9', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '3e739dbb-4c9c-4d4b-b37d-a02ef76897c9', 'content-type': 'application/x-amz-json-1.1', 'content-length': '90', 'date': 'Thu, 01 Jul 2021 18:36:07 GMT'}, 'RetryAttempts': 0}})>
```

I'm not sure *why* it's happening given the defaults should work out to 5 vCPUs and my account limit is 5 vCPUs, and I did check to make sure I didn't have anything else running. Maybe there's some overhead, maybe it's user error.

The fix seems easy enough:

```python
from dask_cloudprovider.aws import FargateCluster
cluster = FargateCluster(n_workers=1, worker_cpu=256, worker_mem=512)
```

but as a brand-new user, this meant I had to spend the rest of my day learning about AWS instead of continuing with the tutorial.

Searching for that error just brings up this [thread](https://forums.aws.amazon.com/thread.jspa?threadID=335825) where a ton of other users collectively fail to figure out which service quota limit it is they have to ask for an increase in, so it would also be nice if the documentation pointed new users in the right direction though I get it if that's outside the scope of the dask project.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.