aws / aws/aws-node-termination-handler
Queue Processor deletes an SQS message after a taint error
- Lingua principale
- Go
- Stelle
- 1.8k
- Fork
- 286
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Problem
When Queue Processor handles an `EC2 Instance Rebalance Recommendation`, a failure to add a taint to the Node is not treated as an error.
The taint error is logged, but node draining and SQS message deletion continue.
The SQS message cannot be retried.
## Related
#1279 updated the Queue Processor Spot interruption handler to return taint errors so that the SQS message can be retried.
The `EC2 Instance Rebalance Recommendation` handler does not propagate the error in the same way.
## Steps to reproduce
1. Set up NTH in Queue Processor mode, two EKS Nodes, an SQS queue, and a test Pod.
2. Configure a `ValidatingAdmissionPolicy` that rejects taint updates by NTH for the target Node.
3. Send an `EC2 Instance Rebalance Recommendation` event for the target Node directly to the SQS queue.
4. Check the NTH logs, target Node, test Pod, and SQS message.
No EC2 instance was terminated. The event was sent directly to the test SQS queue in the EC2 EventBridge event format.
## Expected outcome
When adding the taint fails, Queue Processor should stop processing the event.
- The Node should not be cordoned or drained.
- The SQS message should not be deleted.
- The same SQS message should be received again after the visibility timeout.
## Actual outcome
- The taint update was rejected.
- Node draining continued.
- The test Pod moved to the other Node.
- The SQS message could not be received again after a 30-second visibility timeout.
## Application Logs
```text
Unable to taint node with taint
aws-node-termination-handler/rebalance-recommendation
Draining the node node_name=
evicting pod nth-verification/test-app
```
## Environment
- NTH revision: 62a8761d812b331620a0f006b55f70bcadb0462e (built locally)
- NTH mode: Queue Processor
- Worker Node OS / architecture: Amazon Linux 2023 / amd64
- Kubernetes version: EKS 1.34
- Installation method: Helm chart from the same revision
- SQS visibility timeout: 30 seconds
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia con il Queue Processor EC2 Instance Rebalance Recommendation handler e confronta la sua gestione degli errori con quella dello Spot interruption handler modificato in #1279. Riproduci il rifiuto del taint, se possibile, quindi verifica che un errore di taint interrompa il cordoning e il draining e lasci il messaggio SQS disponibile per un nuovo tentativo dopo il visibility timeout.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- aws, go, kubernetes
- Ambito
- backend, cloud, devops
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 68/100