etcd-io / etcd-io/etcd-operator
auto cert provider cannot satisfy clientCertAuth: SelfCert certs are ServerAuth-only and self-rooted, wedging multi-member TLS clusters
- Dominant language
- Go
- Stars
- 164
- Forks
- 72
- Avg merge
- 22h 46m
- Merged PRs (30d)
- 34
Description
## Summary
The `auto` certificate provider is structurally incompatible with `clientCertAuth: true` (the CRD default): the operator's self-minted client certificate can never be verified by etcd, so a multi-member auto-TLS cluster wedges at one member.
## Root cause
The auto provider mints each certificate (server, peer, operator-client) via an independent `transport.SelfCert` call, and SelfCert produces certificates that are:
1. **Self-rooted** — each certificate is its own trust root, minted into its own secret. etcd's `--trusted-ca-file` points at the *server* secret's `ca.crt`, which can never verify the operator's client certificate (a different self-signed root).
2. **ServerAuth-only** — `transport.SelfCert` sets `ExtKeyUsage: [ServerAuth]` unless `additionalUsages` are passed (client/pkg/transport/listener.go:278 @ v3.6.12), and the provider passes none. Even with a shared root, the client cert would fail EKU validation for client auth.
## Observed behavior
With `spec.tls: {peer: {provider: auto}, client: {provider: auto}}` (clientCertAuth defaulted true), pod 0 starts and runs, but:
```
etcd: "rejected connection on client endpoint" ... "error":"remote error: tls: bad certificate" (operator dials)
etcd: "rejected connection on client endpoint" ... "error":"tls: failed to verify certificate: x509: certificate specifies an incompatible key usage" (etcd's own loopback client)
```
The operator's health checks against member 0 fail, reconciliation never scales past 1 replica, and `test/e2e TestClusterAutoCertCreation` times out waiting for 3-replica readiness. Because `waitForStatefulSetReady` blocks the (single) reconcile worker, the wedged cluster then starves every other EtcdCluster in the manager.
Reproduced identically at the current tip of the TLS-independence branch **and at its pre-reshape base** — the defect is inherent to the auto provider's cert model (which predates all of the TLS-consumption work) and only becomes reachable once certificates are actually mounted and enforced, i.e. exactly what #376 wires up. It has not been caught in CI because PR runs are ok-to-test-gated.
## Suggested direction
The auto provider needs a per-cluster CA model: mint one self-signed CA per cluster (or per surface), sign the server/peer/operator-client certificates from it with both ServerAuth and ClientAuth EKUs, and distribute that CA as the trust anchor. `transport.SelfCert` accepts `additionalUsages ...x509.ExtKeyUsage`, which covers the EKU half, but the shared-root half needs the provider to stop minting independent self-signed leaves.
Until then, `provider: auto` is only usable with `clientCertAuth: false` (server-only TLS), which may be worth documenting or enforcing.
Contributor guide
Research direction
Start with the auto certificate provider and client/pkg/transport/listener.go around line 278 to understand the current certificate usages. Run test/e2e TestClusterAutoCertCreation and inspect waitForStatefulSetReady to reproduce the one-member stall. Done means auto certificates share an appropriate per-cluster trust anchor, support the required client and server usages, and the multi-replica readiness test passes without blocking other clusters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- authentication, backend, security
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100