[ECS] [request]: NLB forwards to "unused" targets
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
**Tell us about your request**
After updating an NLB listener to point to another target group, the NLB keeps sending requests (new connection per request, but probably same flow) to the previous targets for ~75 seconds.
**Which service(s) is this request for?**
NLB, ECS, Fargate
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I'm using CodeDeploy + ECS blue/green deploys, where a new task set is registered with the green target group, and the test and production NLB listener rules are updated to point to the new green target group. This leaves the previous target group with all targets in state "unused".
At that point, I would expect no more requests to the old targets, but connections are still sent to them for ~75 seconds.
This seems to be related to #469 and a general lag in how the NLB interacts with the target group and health checks.
Issuing a rollback using the CodeDeploy+ECS blue/green deployment, is also giving me an outage: The NLB works ok for 10-20 seconds, then starts rejecting connections (TCP reset), then starts timing out, and only 30+ seconds later it starts replying correctly. I'm still researching this.
**Additional context**
I can only speculate that this is related to how the NLB handles traffic flows, and the flow cache is not flushed upon changes to the listener rule and/or target group targets.
Contributor guide
Research direction
Start by reviewing the behavior described in issue #469 and reproducing an ECS blue/green deployment with CodeDeploy, an NLB, and switched target groups. Observe listener changes, target states, connection resets, and timeouts during deployment and rollback; done means determining whether the delayed traffic and outage are expected NLB flow behavior or a service defect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100