dask / dask/dask-cloudprovider
Dask config size limitation in EC2Cluster
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
It seems there is a 16kb limit on the amount of user_data that can be passed to an EC2 instance when starting up.
We serialize the local Dask config and pass it to the scheduler and workers via the user_data.
https://github.com/dask/dask-cloudprovider/blob/da454827b88c2f0f0d06af07e02d5d1580a4c366/dask_cloudprovider/generic/vmcluster.py#L33-L35
Depending on what config the user has locally this can tip us over the limit and result in the AWS API rejecting the instance creation call.
```python-traceback
botocore.exceptions.ClientError: An error occurred (InvalidParameterValue) when calling the RunInstances operation: User data is limited to 16384 bytes
```
Contributor guide
Assessment
This issue has not been assessed yet.