aws / aws/containers-roadmap

[EKS] [request]: Configuration surface for the node monitoring agent on EKS Auto Mode

Open
#2,872 0 comments 0 reactions 0 assignees View on GitHub
EKS EKS Auto Mode Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

> **Community Note**
> - Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
> - Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
> - If you are interested in working on this issue or have submitted a pull request, please leave a comment

## Tell us about your request

Please give EKS Auto Mode a supported way to configure the node monitoring agent (NMA), in particular its interface-exclusion settings (`excludedInterfaceNameRegexps`) and/or per-condition repair behavior. Today the NMA is effectively unconfigurable on Auto Mode, so when it raises a false-positive node condition there is no supported lever to suppress it.

## Which service(s) is this request for?

EKS (Auto Mode), Bottlerocket

## Tell us about the problem you are trying to solve

On EKS Auto Mode, the agent that evaluates node conditions is baked into the node image and reads its configuration from a file that Auto Mode never writes, so it always runs on the values compiled into it. The separate node monitoring agent add-on does expose configuration, but its DaemonSet is excluded from Auto Mode nodes by design, so installing and configuring the add-on has no effect on Auto Mode nodes. There is no NodeClass field, no Bottlerocket setting, and no userData passthrough that reaches the agent either.

This creates issues when the agent raises a false-positive `InterfaceNotUp` finding on a harmless, administratively-down virtual interface, it sets `NetworkingReady=False`. Node auto-repair treats that as a node-level networking failure and terminates an otherwise-healthy node roughly 30 minutes later, bypassing PodDisruptionBudgets. On a cluster running a GRE-based overlay this reproduces reliably, because loading the `ip_gre` kernel module makes the kernel create admin-down fallback devices (`gre0`, `gretap0`, `erspan0`) that the agent then flags.

AWS confirmed one instance of this as a product defect and addressed it by adding the kernel tunnel fallback devices to the agent's compiled-in default exclusion list ([[aws/eks-node-monitoring-agent#238](https://github.com/aws/eks-node-monitoring-agent/pull/238)](https://github.com/aws/eks-node-monitoring-agent/pull/238)). That is appreciated, but it is device-specific. Because the agent remains unconfigurable on Auto Mode, any future false positive on a different virtual device would be equally impossible for a customer to suppress, and the only remedy would again be an unsupported workaround.

## Are you currently working around this issue?

Yes. We run a privileged DaemonSet that sets `net.core.fb_tunnels_only_for_init_net=2` on every node so the kernel never creates the fallback devices, which removes the trigger at its source. It works, but it means running a privileged, kernel-parameter-mutating pod on every node purely to keep Auto Mode nodes from being terminated over a cosmetic interface, which we would rather not do.

## Additional context

Any one of the following would fix the issue:

- A Bottlerocket `settings` key for the node monitoring agent, consistent with how `settings.kernel.sysctl` already renders `/etc/sysctl.d` on these same nodes. The boot-time render mechanism already works on Auto Mode; there is simply no key for the agent.
- A NodeClass field to pass agent configuration (for example, `excludedInterfaceNameRegexps`).
- Extending `nodeRepairConfigOverrides` (per-condition repair suppression keyed on `nodeMonitoringCondition` / `nodeUnhealthyReason`), which already exists for managed node groups, to Auto Mode.

Related AWS Support case: 178845006406731

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the mentioned NodeClass, Bottlerocket settings, nodeRepairConfigOverrides, and node monitoring agent configuration surfaces. Compare the existing settings.kernel.sysctl rendering and the managed node group repair override behavior. Done means defining and documenting a supported Auto Mode configuration path for the requested exclusions or repair behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.