[EKS] [bug]: extension-apiserver-authentication ConfigMap grows unboundedly — 550 unique CA certificates after ~4 years
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
## [EKS] [bug]: extension-apiserver-authentication ConfigMap grows unboundedly — 550 unique CA certificates after ~4 years
**Cluster age:** June 2022 – present (EKS, Kubernetes 1.34)
### Problem
The `kube-system/extension-apiserver-authentication` ConfigMap has accumulated **550 unique `CN=kubernetes` CA certificates** in its `client-ca-file` key over the ~4-year life of this cluster. The ConfigMap is currently **603KB** — approaching the hard 1MB etcd limit. At the current rate (~40-70 certs/month during active cluster periods), it will exceed the limit within 6-12 months.
```
$ kubectl get cm -n kube-system extension-apiserver-authentication -o yaml | wc -l
10415
$ kubectl get cm -n kube-system extension-apiserver-authentication \
-o jsonpath='{.data.client-ca-file}' | grep -c "BEGIN CERTIFICATE"
550
```
All 550 certificates have:
- Subject: \`CN=kubernetes\`
- 10-year validity (2022→2032 through 2026→2036)
- Unique fingerprints (cryptographically distinct key pairs)
### Root Cause
In **kubernetes/kubernetes#82429** (fixed by PR #82705, merged Nov 2019), the controller that publishes `extension-apiserver-authentication` was changed from "last writer wins" to "union" semantics. The controller (`cluster_authentication_trust_controller.go`) now:
1. Deduplicates via exact byte comparison (`reflect.DeepEqual` on raw DER)
2. Prunes certificates that are more than 5 minutes past their `NotAfter` date
3. Deletes and rebuilds the ConfigMap if it exceeds 1MB
In a self-managed cluster (kubeadm), the cluster CA is generated once at init and reused forever — so deduplication works and the ConfigMap stays small.
**In EKS, the managed control plane generates a fresh CA certificate (new key pair, new serial number) for each API server pod replacement/restart**, rather than reusing a stable cluster CA. This means:
- Each control plane event adds ~3 new certs (one per HA API server node) that are byte-distinct from all prior certs → dedup has no effect
- All certs have 10-year validity → expiry pruning never triggers
- The ConfigMap grows without bound
### Accumulation data (this cluster)
| Year-Month | Certs added |
|------------|-------------|
| 2022-Jun | 7 |
| 2022-Jul | 7 |
| ... | ... |
| 2026-Jan | 45 |
| 2026-Feb | 69 |
| 2026-Mar | 21 (so far) |
| **Total** | **550** |
(Full table available on request)
### Impact
When the ConfigMap hits 1MB, the upstream controller deletes it entirely and rebuilds from scratch. During the brief window between deletion and rebuild, extension API servers (metrics-server, cert-manager webhooks, aggregated APIs) cannot read the configmap, potentially causing:
- Metrics unavailability
- Webhook failures (blocking pod scheduling/creation)
- Extension API unavailability
### Reproduction
```bash
# Count accumulated certificates
kubectl get cm -n kube-system extension-apiserver-authentication \
-o jsonpath='{.data.client-ca-file}' | grep -c "BEGIN CERTIFICATE"
# Check ConfigMap size
kubectl get cm -n kube-system extension-apiserver-authentication -o json | wc -c
# Verify all certificates are unique (no dedup possible)
kubectl get cm -n kube-system extension-apiserver-authentication \
-o jsonpath='{.data.client-ca-file}' | python3 -c "
import sys, subprocess, re
data = sys.stdin.read()
certs = re.findall(r'-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----', data, re.DOTALL)
seen = set()
for cert in certs:
result = subprocess.run(['openssl', 'x509', '-fingerprint', '-sha256', '-noout'],
input=cert.strip(), capture_output=True, text=True)
seen.add(result.stdout.strip())
print(f'Total: {len(certs)}, Unique fingerprints: {len(seen)}')
"
```
### Suggested Resolution
One or more of:
1. **EKS should reuse a stable proxy client CA** across control plane pod replacements rather than generating new CA certs each time — this is the standard behavior in self-managed clusters and would make the upstream dedup effective.
2. **EKS could periodically prune** the \`client-ca-file\` bundle to retain only the most recently issued certificates (e.g., last 30 days), since clients already trust the current CA and old ones are not needed for ongoing TLS.
3. **Document this behavior** so operators are aware and can implement their own cleanup (e.g., a monthly CronJob that trims the ConfigMap).
### References
- [kubernetes/kubernetes#82429](https://github.com/kubernetes/kubernetes/issues/82429) — original issue describing "last one wins" behavior
- [kubernetes/kubernetes PR #82705](https://github.com/kubernetes/kubernetes/pull/82705) — the union semantics controller that assumed short-lived certs or byte-identical reuse
- [Upstream controller source](https://github.com/kubernetes/kubernetes/blob/master/pkg/controlplane/controller/clusterauthenticationtrust/cluster_authentication_trust_controller.go)
Contributor guide
Research direction
Start with the upstream cluster_authentication_trust_controller.go source and the referenced Kubernetes issue #82429 and PR #82705; reproduce the ConfigMap certificate count and size with the provided kubectl commands. Done requires a confirmed EKS-side cause and an agreed resolution among stable CA reuse, safe pruning, or operator documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100