Automatically redeploy when image sha changes
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
I would like to re-open a discussion on this https://github.com/aws/amazon-ecs-agent/issues/189
It would be great if ecs agent could automatically keep the container running on the latest image
This would allow for the infrastructure team to pin their service deploys to the `latest` tag
Then, when the service developers push a new release, they push a semver'd image, along with overwriting the existing `latest` tagged image with the new one
As @euank pointed out 6 years ago, in the thread linked, such rolling deploys were normally done in the build pipeline. However, I think the devops landscape has slightly shifted since then.
Our company, for one, is succesfully managing all of its multi-aws-account, multi-region, multi-environment infrastructure in terraform, in a single repository. New services are deployed to ECS or Lambda by writing a terraform module for the service, which the ci/cd pipeline then automatically deploys, and keeps up to date whenever any changes to it are made.
This works very well and creates a clear boundary between service code, and infrastructure. The code of each service component sits in their own repositories, and the associated ci/cd pipelines are only responsible for testing the code and publishing a build, closely reflecting the practices of open-source. In turn, it is then up to the infrastructure team to take said build, and deploy it as a service, into as many environments or regions as they see fit, just like they would with any other OSS project.
However, the above outlined workflow creates a friction point: after the software dev team make a new release, they have to raise another pull request into the infrastructure repo to update all instances of the running software, to the latest release. They have to do this first for staging, then for production.
It's not a massive hassle, takes 2-3 minutes tops, but ends up wasting many man hours when you do many deploys for many services many times a day.
So, naturally, like all other teams, we would like to have a system in place that updates running services to the latest build when a new release is made. Historically, this would be done in the same ci/cd pipeline triggered by a code merge/release in the service repo. This however causes a few problems:
1. CI/CD pipeline of each service requires code to deploy new task definitions. For that it needs to know where the service is deployed, which may be in multiple regions, as mutliple ecs services, etc. Where is the source of truth for this information? How does the pipeline access this source of truth? What kind of permissions does it need to access this information?
2. CI/CD pipeline requires broad AWS permissions for doing the aforementioned deploys, into multiple aws accounts, multiple regions and multiple services, which poses security implications, overcomplicates any permission provisioning practices, and causes a lot of management overhead as well as auditability problems
3. CI/CD pipeline touches resources managed by the IAC tools (terraform/cloudformation), e.g., writing new task definitions
All of the aforementioned complexity can be avoided if the service hosting platform (ECS in this case), would support keeping the containers running on the latest version of an image, then having the infrastructure team pin to the `latest` tag in ecr.
tl;dr: a lot of complexity in the form on bespoke rolling deploy CI/CD pipelines would be saved if the ecs-agent had an option to have containers auto-redeploy when source image is updated. Pinning your task definition to an image tagged with `latest` in ecr means you simply need to push a new image to the `latest` tag on ecr, and all your services using that images will start running it without any extra work required
Thank you
Contributor guide
Assessment
This issue has not been assessed yet.