aws / aws/containers-roadmap

[Fargate] [request]: Fargate Spot Connection Draining event stopCode: "TerminationNotice"

Open
#1,565 1 comment 8 reactions 0 assignees View on GitHub
Docs ECS Fargate Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
AWS FARGATE_SPOT events are not working as described in documentation which causes issues in FARGATE_SPOT connection draining via EventBridge and Lambda function

**Which service(s) is this request for?**
Fargate, ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
In multiple AWS Documentation resources:
- https://aws.amazon.com/blogs/containers/graceful-shutdowns-with-ecs/
- https://aws.amazon.com/blogs/compute/deep-dive-into-fargate-spot-to-run-your-ecs-tasks-for-up-to-70-less/
- https://github.com/aws-samples/ecs-fargate-drain-function

is written that in order to handle FARGATE_SPOT task termination we should listen for events with `detail.stopCode == "TerminationNotice"` property, but in reality `stopCode` is not populated, only `detail.stopReason: "Your Spot Task was interrupted."` comes populated in event .

I've just gotten answer from AWS Support team stating it is an issue on AWS side:

```
Please note that this is a known issue/bug and according to the internal team
the 'stopCode' can be empty when 'stopReason' is 'Your Spot Task was interrupted'.
Moving forward the team has informed that for handling FARGATE_SPOT termination,
the 'stoppedReason' should only be used instead of 'stopCode'.

I would also like to inform you that our internal team agreed that this indeed contradicts the AWS
documentation and the team is working internally to make the necessary changes to the documentation.
Unfortunately, I'll not be able to provide an ETA when it would be reflected.
```

Event example:

```
{
"version": "0",
"id": "78a67cf9-1111-3333-2222-9bfa12345678",
"detail-type": "ECS Task State Change",
"source": "aws.ecs",
"account": "account",
"time": "2021-11-18T14:13:23Z",
"region": "eu-central-1",
"resources": [
"arn:aws:ecs:eu-central-1:account:task/prod-cluster/12345678acf74b8b91c0db5ba3e81234"
],
"detail": {
"clusterArn": "arn:aws:ecs:eu-central-1:account:cluster/prod-cluster",
"containers": [
{
"containerArn": "arn:aws:ecs:eu-central-1:account:container/prod-cluster/12345678acf74b8b91c0db5ba3e81234/7f9efa07-1111-2222-96d2-123456781234",
"lastStatus": "RUNNING",
"name": "core-container-prod",
"taskArn": "arn:aws:ecs:eu-central-1:account:task/prod-cluster/12345678acf74b8b91c0db5ba3e81234",
"networkInterfaces": [
{
"attachmentId": "ba731d36-1598-4f83-b465-a3d50dd1ca10",
"privateIpv4Address": "100.3.117.70"
}
],
"cpu": "0"
}
],
"createdAt": "2021-11-15T23:08:54.832Z",
"launchType": "FARGATE",
"cpu": "2048",
"memory": "4096",
"desiredStatus": "STOPPED",
"group": "service:core-service-prod",
"lastStatus": "DEACTIVATING",
"overrides": {
"containerOverrides": [
{
"name": "core-container-prod"
}
]
},
"attachments": [
{
"id": "6f3aa01c-0786-4a10-9468-efbea6c4f848",
"type": "elb",
"status": "ATTACHED"
},
{
"id": "ba731d36-1598-4f83-b465-a3d50dd1ca10",
"type": "eni",
"status": "ATTACHED",
"details": [
{
"name": "subnetId",
"value": "subnet-111120cdd04192222"
},
{
"name": "networkInterfaceId",
"value": "eni-1111f86ad96b22222"
},
{
"name": "macAddress",
"value": "0a:88:59:88:a6:88"
},
{
"name": "privateDnsName",
"value": "ip-100-3-117-70.eu-central-1.compute.internal"
},
{
"name": "privateIPv4Address",
"value": "100.3.117.70"
}
]
},
{
"id": "12345678-9496-4900-8888-1fb592081234",
"type": "elb",
"status": "ATTACHED"
}
],
"connectivity": "CONNECTED",
"connectivityAt": "2021-11-15T23:08:58.236Z",
"pullStartedAt": "2021-11-15T23:09:19.035Z",
"startedAt": "2021-11-15T23:10:27.583Z",
"startedBy": "ecs-svc/6522240927423895723",
"stoppingAt": "2021-11-18T14:13:23.683Z",
"pullStoppedAt": "2021-11-15T23:09:52.079Z",
"stoppedReason": "Your Spot Task was interrupted.",
"updatedAt": "2021-11-18T14:13:23.683Z",
"taskArn": "arn:aws:ecs:eu-central-1:account:task/prod-cluster/123456789012345",
"taskDefinitionArn": "arn:aws:ecs:eu-central-1:account:task-definition/core-task-prod:23",
"version": 5,
"platformVersion": "1.4.0"
}
}

```

**Are you currently working around this issue?**
I will just watch `detail.stopReason: "Your Spot Task was interrupted.",` and try to deregister tasks from target groups via lambda function.

**Additional context**
This issue is related with https://github.com/aws/containers-roadmap/issues/797 but tackles different thing, as here we are speaking that AWS ECS Events are not working by specification.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the AWS ECS and Fargate documentation links with the supplied EventBridge event example, focusing on stopCode, stopReason, and stoppedReason. Confirm the expected Fargate Spot termination behavior and review the related issue 797. Done means the service behavior and documentation consistently identify how connection draining should detect termination.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.