dask / dask/dask-cloudprovider

Allow specification of secrets in scheduler and worker container definitions for ECSCluster / FargateCluster

Open
#332 3 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted provider/aws/ecs
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

AWS allows you to [specify 'secrets'](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html) in container definitions. This avoids having to call secrets manager directly in the application code.

The secrets key-value pair would need to be added to this block below (and again for the worker) based on init parameters.

```python
containerDefinitions=[
{
"name": "dask-scheduler",
"image": self.image,
"cpu": self._scheduler_cpu,
"memory": self._scheduler_mem,
"memoryReservation": self._scheduler_mem,
"essential": True,
"command": [
"dask-scheduler",
"--idle-timeout",
self._scheduler_timeout,
]
+ (
list()
if not self._scheduler_extra_args
else self._scheduler_extra_args
),
"ulimits": [
{
"name": "nofile",
"softLimit": 65535,
"hardLimit": 65535,
},
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": ecs.meta.region_name,
"awslogs-group": self.cloudwatch_logs_group,
"awslogs-stream-prefix": self._cloudwatch_logs_stream_prefix,
"awslogs-create-group": "true",
},
},
"mountPoints": self._mount_points
if self._mount_points and self._mount_volumes_on_scheduler
else [],
}
],
```

I envisage reusing the same pattern used by `_scheduler_extra_args`:

```python
"secrets": []
+ (
list()
if not self._scheduler_extra_secrets
else self._scheduler_extra_secrets
)
```

**N.B.** there are other key-value pairs that it could be useful to specify, such as ['environment'](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html) for defining environment variables inline. These could be added in the same way, or better yet, in a more generic way that allows you to specify arbitrary containerDefinitions kwargs.

I'd be happy to provide a PR for either approach.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.