etcd-io / etcd-io/etcd-operator

auto cert provider cannot satisfy clientCertAuth: SelfCert certs are ServerAuth-only and self-rooted, wedging multi-member TLS clusters

Open
#420 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
164
Forks
72
Avg merge
22h 46m
Merged PRs (30d)
34

Description

## Summary

The `auto` certificate provider is structurally incompatible with `clientCertAuth: true` (the CRD default): the operator's self-minted client certificate can never be verified by etcd, so a multi-member auto-TLS cluster wedges at one member.

## Root cause

The auto provider mints each certificate (server, peer, operator-client) via an independent `transport.SelfCert` call, and SelfCert produces certificates that are:

1. **Self-rooted** — each certificate is its own trust root, minted into its own secret. etcd's `--trusted-ca-file` points at the *server* secret's `ca.crt`, which can never verify the operator's client certificate (a different self-signed root).
2. **ServerAuth-only** — `transport.SelfCert` sets `ExtKeyUsage: [ServerAuth]` unless `additionalUsages` are passed (client/pkg/transport/listener.go:278 @ v3.6.12), and the provider passes none. Even with a shared root, the client cert would fail EKU validation for client auth.

## Observed behavior

With `spec.tls: {peer: {provider: auto}, client: {provider: auto}}` (clientCertAuth defaulted true), pod 0 starts and runs, but:

```
etcd: "rejected connection on client endpoint" ... "error":"remote error: tls: bad certificate" (operator dials)
etcd: "rejected connection on client endpoint" ... "error":"tls: failed to verify certificate: x509: certificate specifies an incompatible key usage" (etcd's own loopback client)
```

The operator's health checks against member 0 fail, reconciliation never scales past 1 replica, and `test/e2e TestClusterAutoCertCreation` times out waiting for 3-replica readiness. Because `waitForStatefulSetReady` blocks the (single) reconcile worker, the wedged cluster then starves every other EtcdCluster in the manager.

Reproduced identically at the current tip of the TLS-independence branch **and at its pre-reshape base** — the defect is inherent to the auto provider's cert model (which predates all of the TLS-consumption work) and only becomes reachable once certificates are actually mounted and enforced, i.e. exactly what #376 wires up. It has not been caught in CI because PR runs are ok-to-test-gated.

## Suggested direction

The auto provider needs a per-cluster CA model: mint one self-signed CA per cluster (or per surface), sign the server/peer/operator-client certificates from it with both ServerAuth and ClientAuth EKUs, and distribute that CA as the trust anchor. `transport.SelfCert` accepts `additionalUsages ...x509.ExtKeyUsage`, which covers the EKU half, but the shared-root half needs the provider to stop minting independent self-signed leaves.

Until then, `provider: auto` is only usable with `clientCertAuth: false` (server-only TLS), which may be worth documenting or enforcing.

Contributor guide

Open the contributing guide

Research direction

Start with the auto certificate provider and client/pkg/transport/listener.go around line 278 to understand the current certificate usages. Run test/e2e TestClusterAutoCertCreation and inspect waitForStatefulSetReady to reproduce the one-member stall. Done means auto certificates share an appropriate per-cluster trust anchor, support the required client and server usages, and the multi-replica readiness test passes without blocking other clusters.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
authentication, backend, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.