dask / dask/dask-cloudprovider

Dask config size limitation in EC2Cluster

Open
#249 11 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

It seems there is a 16kb limit on the amount of user_data that can be passed to an EC2 instance when starting up.

We serialize the local Dask config and pass it to the scheduler and workers via the user_data.

https://github.com/dask/dask-cloudprovider/blob/da454827b88c2f0f0d06af07e02d5d1580a4c366/dask_cloudprovider/generic/vmcluster.py#L33-L35

Depending on what config the user has locally this can tip us over the limit and result in the AWS API rejecting the instance creation call.

```python-traceback
botocore.exceptions.ClientError: An error occurred (InvalidParameterValue) when calling the RunInstances operation: User data is limited to 16384 bytes
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.