aws / aws/containers-roadmap

[EKS] [eks-node-monitoring-agent]: Automate collection of node logs via EKS Node Monitoring Agent

Open
#2,762 0 comments 2 reactions 0 assignees View on GitHub
EKS Nodes Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

### Summary

Request to enhance the Node Monitoring Agent (NMA) with automatic log collection and S3 upload capability when nodes are being terminated. This addresses a critical pain point where node logs become inaccessible after termination, significantly impacting troubleshooting capabilities.

### Problem Statement

We frequently encounter difficulty collecting logs from EKS nodes due to nodes being terminated or deleted before log collection can be completed. This issue appears in cases related to node issues and manifests in several scenarios:
```
Autoscaler-initiated terminations: Nodes terminated by cluster autoscaler during scale-down events
Automatic node replacement: Nodes replaced due to health checks or maintenance operations
Hardware failure terminations: Unexpected node termination due to underlying infrastructure issues
Quick termination cycles: Rapid node lifecycle changes in dynamic environments
```
Currently, operators must manually create NodeDiagnostic resources and pre-signed S3 URLs before termination occurs, which is often impractical in automated environments.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

Enhance NMA to automatically detect impending node termination and proactively collect and upload logs to S3 without manual intervention.

**Are you currently working around this issue?**
How are you currently solving this problem?

**Additional context**
Anything else we should know?

**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the EKS Node Monitoring Agent's handling of impending node termination and the existing NodeDiagnostic and S3 workflow. Done means the agent can detect relevant termination scenarios, collect node logs, and upload them to S3 without manually creating resources or pre-signed URLs.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.