AI Gateway webhook cert becomes untrusted after pod rescheduling, blocking envoy proxy creation

Problem

When the AI Gateway controller pod is rescheduled (e.g. after a node replacement or during routine cluster maintenance), the envoy proxy data plane pod cannot be created. The API server rejects the pod because the AI Gateway mutating webhook's TLS certificate is no longer trusted.

ReplicaSet events show:

failed calling webhook "ai-gateway-controller.envoy-ai-gateway-system.svc.cluster.local":
x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error"
while trying to verify candidate authority certificate "ai-gateway-controller-ca")

The envoy proxy deployment stays at 0/1, NLB target groups have zero registered targets, and all external connectivity is lost.

Root cause (suspected)

The AI Gateway controller generates a self-signed CA and serving cert at startup, then patches the MutatingWebhookConfiguration with the CA bundle. When the controller pod is rescheduled to a different node, the new pod generates a new CA but the webhook config may retain the old CA bundle until the controller finishes its startup reconcile. If that reconcile is slow or fails silently, the API server continues using the stale CA and rejects all pod creation matching the webhook selector - including the envoy proxy pod.

Impact

Complete loss of external connectivity. No model traffic can flow until the proxy pod is recreated.

Current workaround

kubectl rollout restart deploy -n envoy-ai-gateway-system ai-gateway-controller
kubectl rollout status deploy -n envoy-ai-gateway-system ai-gateway-controller
# Envoy proxy pod recovers within ~30 seconds

Investigation needed

• Is this an upstream Envoy AI Gateway bug? Should it use cert-manager instead of self-signed certs?
• Is there a startup race condition where the controller serves traffic before updating the webhook config?
• Is this reproducible on every reschedule, or only under specific timing conditions (e.g. slow API server)?
• Should we file this upstream at envoyproxy/ai-gateway?

Context

Discovered during the fresh-install validation for #65. Documented in docs/install-production.md section 12.3.

Start with docs/install-production.md section 12.3, then inspect the AI Gateway controller startup and webhook reconciliation behavior during a rollout or pod reschedule. Reproduce the failure while watching ReplicaSet events and the MutatingWebhookConfiguration; done means the certificate remains trusted after rescheduling, or the upstream Envoy AI Gateway issue and required follow-up are documented.