[ECS] [request]: Control scale-down behavior during CodeDeploy blue-green deploy
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
I would like more control over the "Terminate original task set" step of a CodeDeploy blue-green deployment. Specifically, I need the ability to spread task termination over a specified time window, similarly to how we currently have the ability to spread traffic shifting over a time window via deployment configurations.
**Which service(s) is this request for?**
ECS and CodeDeploy
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I have a service which serves long-running WebSocket connections. Setting up a connection involves some relatively expensive operations, so I need to avoid scenarios that trigger thundering herds, where all connected clients get disconnected simultaneously and reconnect. The current behavior of CodeDeploy blue-green deployments is such a trigger, because:
- Traffic shifting does not apply to existing connections, only to new connections which are established during the shifting window.
- Existing connections are disconnected en masse when CodeDeploy terminates the original task set. There's no option available to control this termination other than to wait for some period of time, which doesn't help. I tried waiting to terminate for 10 minutes, and using `UpdateTaskSet` during that interval to scale down the original task set, but it produces the following error:
> Amazon ECS does not support task set management on services where deployments are controlled by AWS CodeDeploy.
The impact of this is that I can't deploy with CodeDeploy during peak traffic without causing a service outage due to the thundering herd of clients reconnecting.
**Are you currently working around this issue?**
I could solve this with an external deployment that calls `UpdateTaskSet` repeatedly, progressively scaling down the original task set over the desired time period. However, I would have to re-implement the rest of blue-green deployments from scratch, and I would lose access to the convenient CodeDeploy deployment UI.
Contributor guide
Research direction
The request names ECS, CodeDeploy, UpdateTaskSet, and the CodeDeploy deployment UI, but no repository files, tests, or implementation entry points. Start by determining whether this roadmap repository has an implementation location for the request and review the reported UpdateTaskSet limitation. Done means CodeDeploy can progressively terminate the original task set over a specified time window without requiring a replacement deployment implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100