aws / aws/containers-roadmap

[EKS] [request]: kube-proxy add-on should roll the DaemonSet when its configuration changes

Open
#2,849 0 comments 13 reactions 0 assignees View on GitHub
EKS EKS Add-Ons Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

When the EKS-managed `kube-proxy` add-on changes the effective kube-proxy configuration — either through the add-on's own `configurationValues`, or through the `kube-proxy-config` ConfigMap that the add-on owns — the add-on should roll the `kube-proxy` DaemonSet so the new configuration actually takes effect.

Concretely, one of:

1. The add-on manager stamps a hash of the rendered config onto the DaemonSet pod template (e.g. `eks.amazonaws.com/kube-proxy-config-hash: `), the standard "config checksum annotation" pattern, so any config change produces a rolling update; or
2. The add-on manager performs an explicit rolling restart of the DaemonSet after it reconciles the ConfigMap; or
3. At minimum, an opt-in add-on setting (e.g. `restartOnConfigChange: true`) plus an explicit statement of the current behaviour in the docs.

**Which service(s) is this request for?**

EKS (managed add-ons: `kube-proxy`)

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

kube-proxy reads its configuration file once, at process start. Upstream has stated that dynamic reload is not going to be supported, and the in-tree config file watcher does not reliably restart the process:

* https://github.com/kubernetes/kubernetes/issues/132877 — kube-proxy doesn't restart on config change
* https://github.com/kubernetes/kubernetes/issues/57144 — updating the kube-proxy ConfigMap doesn't reload/restart kube-proxy
* https://github.com/kubernetes/kubernetes/issues/94022 — global tracking issue for ConfigMap reloads

So on EKS the result is a silent no-op: `UpdateAddon` (or a Terraform `aws_eks_addon` apply) reports success, the `kube-proxy-config` ConfigMap holds the new values, the API and the console show the add-on as `ACTIVE` with the new `configurationValues` — but every running kube-proxy pod is still serving the old configuration, indefinitely, until something unrelated recycles the pods (a node replacement, an add-on version bump, a manual `kubectl rollout restart`).

Why this hurts:

* The change silently does not apply. There is no drift indicator anywhere in the EKS API: desired state and reported state agree, while actual node behaviour does not.
* It affects settings people change precisely because they are having a production problem — `conntrack.maxPerCore` / `conntrack.min` under connection-table exhaustion, `iptables.masqueradeAll` during the IPVS-to-iptables migration, `metricsBindAddress`, log verbosity while debugging. The fix looks applied and the incident continues.
* The blast radius is cluster-wide and node-local: partial rollouts are the norm, because nodes that happen to be replaced later pick up the new config while long-lived nodes keep the old one. Two nodes in the same cluster then run different kube-proxy configurations for weeks, which is very hard to diagnose.
* Every consumer has to reimplement the same restart glue. Terraform users especially: `aws_eks_addon` completes successfully and there is no add-on-native way to express "and roll the DaemonSet", so each module grows a null_resource / Kubernetes Job / checksum-annotation patch of its own.

This is adjacent to, but distinct from, the several open requests to widen the add-on configuration schema — those are about *being able to set* a value, this one is about a value that has been set not being *applied*:

* https://github.com/aws/containers-roadmap/issues/2385 — conntrack settings not in the schema
* https://github.com/aws/containers-roadmap/issues/2826 — `iptables.masqueradeAll` not in the schema
* https://github.com/aws/containers-roadmap/issues/2519 — verbosity not in the schema

Those gaps make this worse, because they push users to patch the `kube-proxy-config` ConfigMap directly, which never triggers a restart either.

**Are you currently working around this issue?**

Yes. We compute an md5 checksum over the add-on version, the add-on `configuration_values`, and our own ConfigMap overlay, and then drive a rolling restart of the `kube-proxy` DaemonSet whenever that checksum changes, from Terraform, after `aws_eks_addon` converges.

Other workarounds seen in the wild: `kubectl -n kube-system rollout restart daemonset/kube-proxy` run by hand after every add-on change, or CI steps that unconditionally restart kube-proxy on every apply (noisy, restarts even when nothing changed).

All of these depend on knowing that the silent no-op exists in the first place. Anyone who does not know it treats the add-on update as complete.

**Additional context**

* The same class of problem applies to any managed add-on whose workload reads config only at startup; CoreDNS is less affected because it watches its Corefile and reloads.
* If AWS considers the current behaviour intentional, documenting it explicitly on the [kube-proxy add-on page](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) — "changing these values does not restart existing pods; run `kubectl rollout restart` to apply" — would already remove most of the surprise.

Contributor guide

Open the contributing guide

Research direction

Start with the EKS kube-proxy add-on documentation page named in the issue and review the linked Kubernetes configuration-reload issues. Compare the three proposed behaviors: a DaemonSet config hash, an explicit rolling restart, or an opt-in setting with documentation. Done means configuration changes are applied to existing kube-proxy pods or the current behavior and required rollout command are clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.