aws / aws/aws-node-termination-handler

Queue Processor deletes an SQS message after a taint error

Open
#1,284 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.8k
Forks
286
PR merge metrics
No merged PRs in 30d

Description

## Problem

When Queue Processor handles an `EC2 Instance Rebalance Recommendation`, a failure to add a taint to the Node is not treated as an error.

The taint error is logged, but node draining and SQS message deletion continue.
The SQS message cannot be retried.

## Related

#1279 updated the Queue Processor Spot interruption handler to return taint errors so that the SQS message can be retried.

The `EC2 Instance Rebalance Recommendation` handler does not propagate the error in the same way.

## Steps to reproduce

1. Set up NTH in Queue Processor mode, two EKS Nodes, an SQS queue, and a test Pod.
2. Configure a `ValidatingAdmissionPolicy` that rejects taint updates by NTH for the target Node.
3. Send an `EC2 Instance Rebalance Recommendation` event for the target Node directly to the SQS queue.
4. Check the NTH logs, target Node, test Pod, and SQS message.

No EC2 instance was terminated. The event was sent directly to the test SQS queue in the EC2 EventBridge event format.

## Expected outcome

When adding the taint fails, Queue Processor should stop processing the event.

- The Node should not be cordoned or drained.
- The SQS message should not be deleted.
- The same SQS message should be received again after the visibility timeout.

## Actual outcome

- The taint update was rejected.
- Node draining continued.
- The test Pod moved to the other Node.
- The SQS message could not be received again after a 30-second visibility timeout.

## Application Logs

```text
Unable to taint node with taint
aws-node-termination-handler/rebalance-recommendation

Draining the node node_name=

evicting pod nth-verification/test-app
```

## Environment

- NTH revision: 62a8761d812b331620a0f006b55f70bcadb0462e (built locally)
- NTH mode: Queue Processor
- Worker Node OS / architecture: Amazon Linux 2023 / amd64
- Kubernetes version: EKS 1.34
- Installation method: Helm chart from the same revision
- SQS visibility timeout: 30 seconds

Contributor guide

Open the contributing guide

Research direction

Start with the Queue Processor EC2 Instance Rebalance Recommendation handler and compare its error handling with the Spot interruption handler changed in #1279. Reproduce the taint rejection if possible, then verify that a taint failure stops cordoning and draining and leaves the SQS message available for retry after the visibility timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
backend, cloud, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.