aws / aws/containers-roadmap

[ECS] [request]: NLB forwards to "unused" targets

Open
#470 7 comments 26 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

**Tell us about your request**
After updating an NLB listener to point to another target group, the NLB keeps sending requests (new connection per request, but probably same flow) to the previous targets for ~75 seconds.

**Which service(s) is this request for?**
NLB, ECS, Fargate

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I'm using CodeDeploy + ECS blue/green deploys, where a new task set is registered with the green target group, and the test and production NLB listener rules are updated to point to the new green target group. This leaves the previous target group with all targets in state "unused".

At that point, I would expect no more requests to the old targets, but connections are still sent to them for ~75 seconds.

This seems to be related to #469 and a general lag in how the NLB interacts with the target group and health checks.

Issuing a rollback using the CodeDeploy+ECS blue/green deployment, is also giving me an outage: The NLB works ok for 10-20 seconds, then starts rejecting connections (TCP reset), then starts timing out, and only 30+ seconds later it starts replying correctly. I'm still researching this.

**Additional context**
I can only speculate that this is related to how the NLB handles traffic flows, and the flow cache is not flushed upon changes to the listener rule and/or target group targets.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the behavior described in issue #469 and reproducing an ECS blue/green deployment with CodeDeploy, an NLB, and switched target groups. Observe listener changes, target states, connection resets, and timeouts during deployment and rollback; done means determining whether the delayed traffic and outage are expected NLB flow behavior or a service defect.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.