[EKS] [request]: Configuration surface for the node monitoring agent on EKS Auto Mode
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
> **Community Note**
> - Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
> - Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
> - If you are interested in working on this issue or have submitted a pull request, please leave a comment
## Tell us about your request
Please give EKS Auto Mode a supported way to configure the node monitoring agent (NMA), in particular its interface-exclusion settings (`excludedInterfaceNameRegexps`) and/or per-condition repair behavior. Today the NMA is effectively unconfigurable on Auto Mode, so when it raises a false-positive node condition there is no supported lever to suppress it.
## Which service(s) is this request for?
EKS (Auto Mode), Bottlerocket
## Tell us about the problem you are trying to solve
On EKS Auto Mode, the agent that evaluates node conditions is baked into the node image and reads its configuration from a file that Auto Mode never writes, so it always runs on the values compiled into it. The separate node monitoring agent add-on does expose configuration, but its DaemonSet is excluded from Auto Mode nodes by design, so installing and configuring the add-on has no effect on Auto Mode nodes. There is no NodeClass field, no Bottlerocket setting, and no userData passthrough that reaches the agent either.
This creates issues when the agent raises a false-positive `InterfaceNotUp` finding on a harmless, administratively-down virtual interface, it sets `NetworkingReady=False`. Node auto-repair treats that as a node-level networking failure and terminates an otherwise-healthy node roughly 30 minutes later, bypassing PodDisruptionBudgets. On a cluster running a GRE-based overlay this reproduces reliably, because loading the `ip_gre` kernel module makes the kernel create admin-down fallback devices (`gre0`, `gretap0`, `erspan0`) that the agent then flags.
AWS confirmed one instance of this as a product defect and addressed it by adding the kernel tunnel fallback devices to the agent's compiled-in default exclusion list ([[aws/eks-node-monitoring-agent#238](https://github.com/aws/eks-node-monitoring-agent/pull/238)](https://github.com/aws/eks-node-monitoring-agent/pull/238)). That is appreciated, but it is device-specific. Because the agent remains unconfigurable on Auto Mode, any future false positive on a different virtual device would be equally impossible for a customer to suppress, and the only remedy would again be an unsupported workaround.
## Are you currently working around this issue?
Yes. We run a privileged DaemonSet that sets `net.core.fb_tunnels_only_for_init_net=2` on every node so the kernel never creates the fallback devices, which removes the trigger at its source. It works, but it means running a privileged, kernel-parameter-mutating pod on every node purely to keep Auto Mode nodes from being terminated over a cosmetic interface, which we would rather not do.
## Additional context
Any one of the following would fix the issue:
- A Bottlerocket `settings` key for the node monitoring agent, consistent with how `settings.kernel.sysctl` already renders `/etc/sysctl.d` on these same nodes. The boot-time render mechanism already works on Auto Mode; there is simply no key for the agent.
- A NodeClass field to pass agent configuration (for example, `excludedInterfaceNameRegexps`).
- Extending `nodeRepairConfigOverrides` (per-condition repair suppression keyed on `nodeMonitoringCondition` / `nodeUnhealthyReason`), which already exists for managed node groups, to Auto Mode.
Related AWS Support case: 178845006406731
Contributor guide
Research direction
Start by reviewing the mentioned NodeClass, Bottlerocket settings, nodeRepairConfigOverrides, and node monitoring agent configuration surfaces. Compare the existing settings.kernel.sysctl rendering and the managed node group repair override behavior. Done means defining and documenting a supported Auto Mode configuration path for the requested exclusions or repair behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100