aws / aws/aws-cdk

(ecs): Drain hook doesn't publish new message to SNS to call lambda function again if tasks are still draining

Open
#20,469 0 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-ecs bug effort/small p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

I am aware that, ECS provides functionality to drain tasks automatically during deployment/instance refresh via creating lifecycle hook, SNS topic and lambda.

From the blog post[[#1]( https://aws.amazon.com/blogs/compute/how-to-automate-container-instance-draining-in-amazon-ecs/)],

"
Auto Scaling groups support lifecycle hooks that can be invoked to allow custom processes to finish before instances launch or terminate. For this example, the lifecycle hook invokes a Lambda function that performs two tasks:

1. Sets the ECS container instance state to DRAINING.
2. Checks if there are any tasks left on the container instance. If there are running tasks still in process of draining, it posts a message to SNS so that the Lambda function is called again.

Lambda repeats step 2 until there are no tasks running on the container instance OR the heartbeat timeout on the lifecycle hook is reached (set to TTL 15 minutes in the sample CloudFormation template), whichever occurs first. Afterward, control is returned to the Auto Scaling lifecycle hook, and the instance terminates.
"

However, I am seeing a slight discrepancy in CDK code. The discrepancy is that if cluster has task running then lambda function is not publishing SNS message and at the end, lambda function time out and task ends up abruptly[[#2](https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-ecs/lib/drain-hook/lambda-source/index.py#L28)].

References:

[1]Blog Post : https://aws.amazon.com/blogs/compute/how-to-automate-container-instance-draining-in-amazon-ecs/

[2]Source code for CDK : https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-ecs/lib/drain-hook/lambda-source/index.py#L28

[3]Source Code for CloudFormation template :https://github.com/aws-samples/ecs-cid-sample/blob/master/cform/ecs.yaml#L479

### Expected Behavior

During instance termination,
1. if ECS cluster has task running in that instance then LifeCycle hook should publish message to SNS which will invoke lambda function.
2. After that if ECS cluster has task running then Lambda function should publishing SNS message so that function triggered again.
3. Once the task is completed or the heartbeat timeout on the lifecycle hook is reached, Instance should be terminated.

### Current Behavior

During instance termination,
1. if ECS cluster has task running in that instance then LifeCycle hook publishes message to SNS which invokes lambda function.
2. After that if ECS cluster has task running then Lambda function wait for some time then checks again for cluster status and repeat this process until Lambda function times out (Max it can wait up to 15 min due to AWS Lambda function hard runtime limit).
3. Once the task is completed/finished or the heartbeat timeout on the lifecycle hook is reached, Instance is getting terminated.

### Reproduction Steps

### Using CDK

1. Create ECS cluster with AutoScaling group (min/max/desired capacity : 1)
```
const cluster = new Cluster(this, `ECSCluster`, {
vpc: vpc
});
const asg = cluster.addCapacity(`ECSClusterCapacity`, {
instanceType: InstanceType.of(
InstanceClass.T2,
InstanceSize.LARGE
),
desiredCapacity: 1,
minCapacity: 1,
maxCapacity: 1,
updateType: undefined,
signals: Signals.waitForAll({timeout: cdk.Duration.minutes(30)}),
taskDrainTime: cdk.Duration.hours(2)
});
```

2. Add EC2 task to that ECS cluster. Task should be running for more than 15 min. (To reproduce this error run task for 2 hours at least).

```
// create task definition
const taskDefinition = new TaskDefinition(this, `EC2TaskDefinition`, {
compatibility: Compatibility.EC2,
networkMode: NetworkMode.AWS_VPC
});

//Add container

const container = taskDefinition.addContainer(`Container`, {
environment: {
...
},
logging: logging,
cpu: 512,
memoryReservationMiB: 1024,
image: image
});

// Create EC2 task
const ec2Service = new ScheduledEc2Task(this, `EC2Task`, {
schedule: Schedule.rate(cdk.Duration.hours(3)),
desiredTaskCount: 1,
scheduledEc2TaskDefinitionOptions: {
taskDefinition: taskDefinition
},
cluster: cluster,
});
```

4. While task is running, do instance refresh. Your job will be terminated abruptly after 15 min (which is after lambda's hard runtime limit).

### Possible Solution

Instead of checking status in same lambda function, we should call the lambda function again.
``` python
while has_tasks(cluster, instance_arn, task_arns):
time.sleep(10)
```
We should have

``` python
if has_tasks(cluster, instance_arn, task_arns):
sns_resp = SNS.publish(TopicArn=event['Records'][0]['Sns']['TopicArn'],
Message=json.dumps(lifecycle_event),
Subject='Publishing SNS msg to invoke Lambda again.')
print('Posted msg %s to SNS topic.' % (sns_resp['MessageId']))
return # return without doing anything

# else, send success signal to autoscaling lifecycle hook.
```

### Additional Information/Context

_No response_

### CDK CLI Version

cdk 1.156.1

### Framework Version

_No response_

### Node.js Version

v14.18.2

### OS

linux

### Language

Typescript

### Language Version

_No response_

### Other information

I am seeing a slight discrepancy in draining lambda code for CDK. The discrepancy is that if cluster has task running then lambda function is not publishing SNS message and at the end, lambda function time out and task ends up abruptly[[#2](https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-ecs/lib/drain-hook/lambda-source/index.py#L28)]

References:

[1]Blog Post : https://aws.amazon.com/blogs/compute/how-to-automate-container-instance-draining-in-amazon-ecs/

[2]Source code for CDK : https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-ecs/lib/drain-hook/lambda-source/index.py#L28

[3]Source Code for CloudFormation template :https://github.com/aws-samples/ecs-cid-sample/blob/master/cform/ecs.yaml#L479

Contributor guide

Open the contributing guide

Research direction

Start with packages/@aws-cdk/aws-ecs/lib/drain-hook/lambda-source/index.py at the referenced line, then compare its behavior with the linked AWS blog post and CloudFormation sample. Verify that a still-draining task causes a new SNS invocation rather than waiting until timeout, and that the lifecycle hook completes only when draining finishes or its heartbeat expires.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.