[EKS] [request]: kube-proxy add-on should roll the DaemonSet when its configuration changes
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
When the EKS-managed `kube-proxy` add-on changes the effective kube-proxy configuration — either through the add-on's own `configurationValues`, or through the `kube-proxy-config` ConfigMap that the add-on owns — the add-on should roll the `kube-proxy` DaemonSet so the new configuration actually takes effect.
Concretely, one of:
1. The add-on manager stamps a hash of the rendered config onto the DaemonSet pod template (e.g. `eks.amazonaws.com/kube-proxy-config-hash: `), the standard "config checksum annotation" pattern, so any config change produces a rolling update; or
2. The add-on manager performs an explicit rolling restart of the DaemonSet after it reconciles the ConfigMap; or
3. At minimum, an opt-in add-on setting (e.g. `restartOnConfigChange: true`) plus an explicit statement of the current behaviour in the docs.
**Which service(s) is this request for?**
EKS (managed add-ons: `kube-proxy`)
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
kube-proxy reads its configuration file once, at process start. Upstream has stated that dynamic reload is not going to be supported, and the in-tree config file watcher does not reliably restart the process:
* https://github.com/kubernetes/kubernetes/issues/132877 — kube-proxy doesn't restart on config change
* https://github.com/kubernetes/kubernetes/issues/57144 — updating the kube-proxy ConfigMap doesn't reload/restart kube-proxy
* https://github.com/kubernetes/kubernetes/issues/94022 — global tracking issue for ConfigMap reloads
So on EKS the result is a silent no-op: `UpdateAddon` (or a Terraform `aws_eks_addon` apply) reports success, the `kube-proxy-config` ConfigMap holds the new values, the API and the console show the add-on as `ACTIVE` with the new `configurationValues` — but every running kube-proxy pod is still serving the old configuration, indefinitely, until something unrelated recycles the pods (a node replacement, an add-on version bump, a manual `kubectl rollout restart`).
Why this hurts:
* The change silently does not apply. There is no drift indicator anywhere in the EKS API: desired state and reported state agree, while actual node behaviour does not.
* It affects settings people change precisely because they are having a production problem — `conntrack.maxPerCore` / `conntrack.min` under connection-table exhaustion, `iptables.masqueradeAll` during the IPVS-to-iptables migration, `metricsBindAddress`, log verbosity while debugging. The fix looks applied and the incident continues.
* The blast radius is cluster-wide and node-local: partial rollouts are the norm, because nodes that happen to be replaced later pick up the new config while long-lived nodes keep the old one. Two nodes in the same cluster then run different kube-proxy configurations for weeks, which is very hard to diagnose.
* Every consumer has to reimplement the same restart glue. Terraform users especially: `aws_eks_addon` completes successfully and there is no add-on-native way to express "and roll the DaemonSet", so each module grows a null_resource / Kubernetes Job / checksum-annotation patch of its own.
This is adjacent to, but distinct from, the several open requests to widen the add-on configuration schema — those are about *being able to set* a value, this one is about a value that has been set not being *applied*:
* https://github.com/aws/containers-roadmap/issues/2385 — conntrack settings not in the schema
* https://github.com/aws/containers-roadmap/issues/2826 — `iptables.masqueradeAll` not in the schema
* https://github.com/aws/containers-roadmap/issues/2519 — verbosity not in the schema
Those gaps make this worse, because they push users to patch the `kube-proxy-config` ConfigMap directly, which never triggers a restart either.
**Are you currently working around this issue?**
Yes. We compute an md5 checksum over the add-on version, the add-on `configuration_values`, and our own ConfigMap overlay, and then drive a rolling restart of the `kube-proxy` DaemonSet whenever that checksum changes, from Terraform, after `aws_eks_addon` converges.
Other workarounds seen in the wild: `kubectl -n kube-system rollout restart daemonset/kube-proxy` run by hand after every add-on change, or CI steps that unconditionally restart kube-proxy on every apply (noisy, restarts even when nothing changed).
All of these depend on knowing that the silent no-op exists in the first place. Anyone who does not know it treats the add-on update as complete.
**Additional context**
* The same class of problem applies to any managed add-on whose workload reads config only at startup; CoreDNS is less affected because it watches its Corefile and reloads.
* If AWS considers the current behaviour intentional, documenting it explicitly on the [kube-proxy add-on page](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) — "changing these values does not restart existing pods; run `kubectl rollout restart` to apply" — would already remove most of the surprise.
Contributor guide
Research direction
Start with the EKS kube-proxy add-on documentation page named in the issue and review the linked Kubernetes configuration-reload issues. Compare the three proposed behaviors: a DaemonSet config hash, an explicit rolling restart, or an opt-in setting with documentation. Done means configuration changes are applied to existing kube-proxy pods or the current behavior and required rollout command are clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100