aws / aws/containers-roadmap

[ECS] [request]: Control scale-down behavior during CodeDeploy blue-green deploy

Open
#969 0 comments 15 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
I would like more control over the "Terminate original task set" step of a CodeDeploy blue-green deployment. Specifically, I need the ability to spread task termination over a specified time window, similarly to how we currently have the ability to spread traffic shifting over a time window via deployment configurations.

**Which service(s) is this request for?**
ECS and CodeDeploy

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I have a service which serves long-running WebSocket connections. Setting up a connection involves some relatively expensive operations, so I need to avoid scenarios that trigger thundering herds, where all connected clients get disconnected simultaneously and reconnect. The current behavior of CodeDeploy blue-green deployments is such a trigger, because:

- Traffic shifting does not apply to existing connections, only to new connections which are established during the shifting window.
- Existing connections are disconnected en masse when CodeDeploy terminates the original task set. There's no option available to control this termination other than to wait for some period of time, which doesn't help. I tried waiting to terminate for 10 minutes, and using `UpdateTaskSet` during that interval to scale down the original task set, but it produces the following error:

> Amazon ECS does not support task set management on services where deployments are controlled by AWS CodeDeploy.

The impact of this is that I can't deploy with CodeDeploy during peak traffic without causing a service outage due to the thundering herd of clients reconnecting.

**Are you currently working around this issue?**
I could solve this with an external deployment that calls `UpdateTaskSet` repeatedly, progressively scaling down the original task set over the desired time period. However, I would have to re-implement the rest of blue-green deployments from scratch, and I would lose access to the convenient CodeDeploy deployment UI.

Contributor guide

Open the contributing guide

Research direction

The request names ECS, CodeDeploy, UpdateTaskSet, and the CodeDeploy deployment UI, but no repository files, tests, or implementation entry points. Start by determining whether this roadmap repository has an implementation location for the request and review the reported UpdateTaskSet limitation. Done means CodeDeploy can progressively terminate the original task set over a specified time window without requiring a replacement deployment implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.