aws / aws/containers-roadmap

[EKS] [bug]: extension-apiserver-authentication ConfigMap grows unboundedly — 550 unique CA certificates after ~4 years

Open
#2,786 1 comment 3 reactions 0 assignees View on GitHub
EKS
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

## [EKS] [bug]: extension-apiserver-authentication ConfigMap grows unboundedly — 550 unique CA certificates after ~4 years

**Cluster age:** June 2022 – present (EKS, Kubernetes 1.34)

### Problem

The `kube-system/extension-apiserver-authentication` ConfigMap has accumulated **550 unique `CN=kubernetes` CA certificates** in its `client-ca-file` key over the ~4-year life of this cluster. The ConfigMap is currently **603KB** — approaching the hard 1MB etcd limit. At the current rate (~40-70 certs/month during active cluster periods), it will exceed the limit within 6-12 months.

```
$ kubectl get cm -n kube-system extension-apiserver-authentication -o yaml | wc -l
10415

$ kubectl get cm -n kube-system extension-apiserver-authentication \
-o jsonpath='{.data.client-ca-file}' | grep -c "BEGIN CERTIFICATE"
550
```

All 550 certificates have:
- Subject: \`CN=kubernetes\`
- 10-year validity (2022→2032 through 2026→2036)
- Unique fingerprints (cryptographically distinct key pairs)

### Root Cause

In **kubernetes/kubernetes#82429** (fixed by PR #82705, merged Nov 2019), the controller that publishes `extension-apiserver-authentication` was changed from "last writer wins" to "union" semantics. The controller (`cluster_authentication_trust_controller.go`) now:

1. Deduplicates via exact byte comparison (`reflect.DeepEqual` on raw DER)
2. Prunes certificates that are more than 5 minutes past their `NotAfter` date
3. Deletes and rebuilds the ConfigMap if it exceeds 1MB

In a self-managed cluster (kubeadm), the cluster CA is generated once at init and reused forever — so deduplication works and the ConfigMap stays small.

**In EKS, the managed control plane generates a fresh CA certificate (new key pair, new serial number) for each API server pod replacement/restart**, rather than reusing a stable cluster CA. This means:
- Each control plane event adds ~3 new certs (one per HA API server node) that are byte-distinct from all prior certs → dedup has no effect
- All certs have 10-year validity → expiry pruning never triggers
- The ConfigMap grows without bound

### Accumulation data (this cluster)

| Year-Month | Certs added |
|------------|-------------|
| 2022-Jun | 7 |
| 2022-Jul | 7 |
| ... | ... |
| 2026-Jan | 45 |
| 2026-Feb | 69 |
| 2026-Mar | 21 (so far) |
| **Total** | **550** |

(Full table available on request)

### Impact

When the ConfigMap hits 1MB, the upstream controller deletes it entirely and rebuilds from scratch. During the brief window between deletion and rebuild, extension API servers (metrics-server, cert-manager webhooks, aggregated APIs) cannot read the configmap, potentially causing:
- Metrics unavailability
- Webhook failures (blocking pod scheduling/creation)
- Extension API unavailability

### Reproduction

```bash
# Count accumulated certificates
kubectl get cm -n kube-system extension-apiserver-authentication \
-o jsonpath='{.data.client-ca-file}' | grep -c "BEGIN CERTIFICATE"

# Check ConfigMap size
kubectl get cm -n kube-system extension-apiserver-authentication -o json | wc -c

# Verify all certificates are unique (no dedup possible)
kubectl get cm -n kube-system extension-apiserver-authentication \
-o jsonpath='{.data.client-ca-file}' | python3 -c "
import sys, subprocess, re
data = sys.stdin.read()
certs = re.findall(r'-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----', data, re.DOTALL)
seen = set()
for cert in certs:
result = subprocess.run(['openssl', 'x509', '-fingerprint', '-sha256', '-noout'],
input=cert.strip(), capture_output=True, text=True)
seen.add(result.stdout.strip())
print(f'Total: {len(certs)}, Unique fingerprints: {len(seen)}')
"
```

### Suggested Resolution

One or more of:

1. **EKS should reuse a stable proxy client CA** across control plane pod replacements rather than generating new CA certs each time — this is the standard behavior in self-managed clusters and would make the upstream dedup effective.

2. **EKS could periodically prune** the \`client-ca-file\` bundle to retain only the most recently issued certificates (e.g., last 30 days), since clients already trust the current CA and old ones are not needed for ongoing TLS.

3. **Document this behavior** so operators are aware and can implement their own cleanup (e.g., a monthly CronJob that trims the ConfigMap).

### References

- [kubernetes/kubernetes#82429](https://github.com/kubernetes/kubernetes/issues/82429) — original issue describing "last one wins" behavior
- [kubernetes/kubernetes PR #82705](https://github.com/kubernetes/kubernetes/pull/82705) — the union semantics controller that assumed short-lived certs or byte-identical reuse
- [Upstream controller source](https://github.com/kubernetes/kubernetes/blob/master/pkg/controlplane/controller/clusterauthenticationtrust/cluster_authentication_trust_controller.go)

Contributor guide

Open the contributing guide

Research direction

Start with the upstream cluster_authentication_trust_controller.go source and the referenced Kubernetes issue #82429 and PR #82705; reproduce the ConfigMap certificate count and size with the provided kubectl commands. Done requires a confirmed EKS-side cause and an agreed resolution among stable CA reuse, safe pruning, or operator documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.