aws / aws/containers-roadmap

[ECS] Blue/green: validate current listener rule weights before shifting traffic

Open
#2,853 0 comments 2 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
For ECS native blue/green deployments, have the deployment controller validate the *current* production listener-rule weighting before it shifts traffic. If the live weighting doesn't match ECS's expected/managed state — for example, production is already pointed at a target group with zero healthy targets — the deployment should halt and surface a clear error/event instead of proceeding with the shift.

**Which service(s) is this request for?**
ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
With ECS native blue/green, the deployment controller manages traffic by adjusting the weights on a production listener rule that forwards across two target groups (`tg-1`, `tg-2`). The controller decides when to shift based on its own internal deployment state and the health of the target (green) revision's tasks — it does not appear to reconcile against the *actual* current weighting of the listener rule.

This becomes a problem when the listener rule is changed out-of-band between deployments:

1. Steady state: `tg-1` holds the running tasks; the production listener rule sends 100% to `tg-1`. No deployment in progress.
2. An out-of-band `ModifyRule` (e.g. a manual change in the console, or any non-ECS actor) sets the rule to 100% → `tg-2`, which has **0 healthy targets**. The ALB immediately returns 503s. This is already an outage, independent of ECS.
3. A new deployment is started.
4. The deployment controller proceeds using its own tracked state. It registers the new revision, and once those tasks are healthy *from ECS's perspective*, it issues `ModifyListener` to shift per its plan (back to `tg-1`).
5. This is where the issue is: `tg-1` is the "source revision" i.e. this is associated with tasks that will be stopped by ECS deployment. The outage persists and there's no signal that the rule had drifted from the managed state.

In short: the controller trusts its own model of "which target group is active" and doesn't verify it against the ALB before acting. A pre-shift validation (and a halt-on-mismatch) would catch out-of-band drift and prevent shifting traffic onto a target group with no healthy targets.

**Are you currently working around this issue?**
Treating the production listener rule as ECS-managed and do not do any out-of-band changes to the listener rules.

**Additional context**
The desired behavior: at deployment start, the controller reads the current production listener-rule weights and compares them to its expected state; on mismatch, it halts the deployment and raises an actionable error/event rather than shifting blindly.

**Attachments**
-

Contributor guide

Open the contributing guide

Research direction

This roadmap issue names no repository files, tests, or implementation entry points. Start by tracing ECS native blue/green deployment handling and the production listener rule's current weights. Done means the deployment detects a mismatch between live and expected weighting, halts traffic shifting, and reports a clear error or event.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.