aws / aws/containers-roadmap

[FARGATE] [request]: Cloudformation support for RunTasks

Open
#1,145 0 comments 25 reactions 0 assignees View on GitHub
Fargate Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

A simple way to run one-time (initialization) ECS tasks during Cloudformation deployment.

**Which service(s) is this request for?**

Fargate, ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

When deploying via Cloudformation, there is support for [defining tasks](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html), but not for running tasks.

*Why would one want to run a task at Cloudformation deployment time?*

Often before [running services](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-service.html), initialization may be needed. Examples:

1. Creating directories on an attached EFS
1. Installing software onto the EFS
1. Creating a database schema
1. Configuring files for services
1. Adding data to a SQS queue

So in a single Cloudformation template, it's not possible to set up infrastructure and deploy services because the initialization steps cannot be performed between the infrastructure creation and the service deployment.

Some initialization tasks must be run serially; one "Job" completes then another "Job" can begin, such as 1 - 4 in the example.
Other one-time tasks are long running such as 5 in the example and don't need to be serialized.

Kubernetes solves this with [init-containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) and manifest files of `kind: Job`. Example:

```yaml
apiVersion: batch/v1
kind: Job
```

**Are you currently working around this issue?**

No, but have seen convoluted methods of trying to solve this issue. Example:

1. https://www.danneh.org/2019/01/triggering-ecs-runtask-aws-cloudformation/
1. https://github.com/aws/containers-roadmap/issues/392
1. https://aws.amazon.com/step-functions/, but not sure how this would work from Cloudformation.

**Additional context**

Potential idea:

Given a task definition such as:
```yaml
EcsTaskDefinitionInitEfs:
Type: "AWS::ECS::TaskDefinition"
DependsOn:
- EfsMountTarget1
- EfsMountTarget2
Properties:
ContainerDefinitions:
-
Command:
- "mkdir"
- "/efs/etc"
- "&&"
- "mkdir"
- "/efs/var"
Essential: true
Image: "busybox"
LinuxParameters:
Capabilities: {}
LogConfiguration:
LogDriver: "awslogs"
Options:
awslogs-group: !Sub "${AWS::StackName}-log-group"
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: "advanced"
MountPoints:
-
SourceVolume: "efs-root"
ContainerPath: "/efs"
ReadOnly: false
Name: "efsinitcontainer"
Privileged: false
PseudoTerminal: false
ReadonlyRootFilesystem: false
Cpu: 512
ExecutionRoleArn: !Sub "arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole"
Family: !Sub "${AWS::StackName}-task-definition-init-efs"
Memory: 1024
NetworkMode: "awsvpc"
RequiresCompatibilities:
- "FARGATE"
Tags:
- Key: Name
Value: !Sub "${AWS::StackName}-init-efs"
Volumes:
-
Name: "efs-root"
EFSVolumeConfiguration:
AuthorizationConfig: {}
FilesystemId:
Ref: EfsFileSystem
RootDirectory: "/"
```

A running of the defined task could be specified by something like this example:

```yaml
EcsTaskRunInitEfs:
Type: "AWS::ECS::TaskRun"
DependsOn:
- EarlierEcsTaskRun
PlatformVersion: 1.4.0
TaskDefinition: EcsTaskDefinitionInitEfs
Tags:
- Key: Name
Value: !Sub "${AWS::StackName}-init-efs-task-run"
```

where `DependsOn` is an optional YAML key for serializing "Jobs".

Contributor guide

Open the contributing guide

Research direction

Start with the AWS::ECS::TaskDefinition and AWS::ECS::Service documentation linked in the issue, then compare the proposed AWS::ECS::TaskRun resource and DependsOn examples with CloudFormation's existing resource model. Done means defining a CloudFormation mechanism for running one-time ECS/Fargate tasks, including dependency ordering and the stated initialization cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.