aws / aws/aws-node-termination-handler

Queue Processor deletes an SQS message after a taint error

Aperta
#1,284 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Go
Stelle
1.8k
Fork
286
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Problem

When Queue Processor handles an `EC2 Instance Rebalance Recommendation`, a failure to add a taint to the Node is not treated as an error.

The taint error is logged, but node draining and SQS message deletion continue.
The SQS message cannot be retried.

## Related

#1279 updated the Queue Processor Spot interruption handler to return taint errors so that the SQS message can be retried.

The `EC2 Instance Rebalance Recommendation` handler does not propagate the error in the same way.

## Steps to reproduce

1. Set up NTH in Queue Processor mode, two EKS Nodes, an SQS queue, and a test Pod.
2. Configure a `ValidatingAdmissionPolicy` that rejects taint updates by NTH for the target Node.
3. Send an `EC2 Instance Rebalance Recommendation` event for the target Node directly to the SQS queue.
4. Check the NTH logs, target Node, test Pod, and SQS message.

No EC2 instance was terminated. The event was sent directly to the test SQS queue in the EC2 EventBridge event format.

## Expected outcome

When adding the taint fails, Queue Processor should stop processing the event.

- The Node should not be cordoned or drained.
- The SQS message should not be deleted.
- The same SQS message should be received again after the visibility timeout.

## Actual outcome

- The taint update was rejected.
- Node draining continued.
- The test Pod moved to the other Node.
- The SQS message could not be received again after a 30-second visibility timeout.

## Application Logs

```text
Unable to taint node with taint
aws-node-termination-handler/rebalance-recommendation

Draining the node node_name=

evicting pod nth-verification/test-app
```

## Environment

- NTH revision: 62a8761d812b331620a0f006b55f70bcadb0462e (built locally)
- NTH mode: Queue Processor
- Worker Node OS / architecture: Amazon Linux 2023 / amd64
- Kubernetes version: EKS 1.34
- Installation method: Helm chart from the same revision
- SQS visibility timeout: 30 seconds

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia con il Queue Processor EC2 Instance Rebalance Recommendation handler e confronta la sua gestione degli errori con quella dello Spot interruption handler modificato in #1279. Riproduci il rifiuto del taint, se possibile, quindi verifica che un errore di taint interrompa il cordoning e il draining e lasci il messaggio SQS disponibile per un nuovo tentativo dopo il visibility timeout.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, go, kubernetes
Ambito
backend, cloud, devops
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
68/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.