ingress-controller webhook certificate rendering causes perpetual Argo CD drift

Affected chart

• api7/api7-ingress-controller 0.1.25
• Current main has the same template behavior

Problem

The Ingress Controller chart produces different webhook certificate data every time it is rendered without cluster access. This leaves an Argo CD application permanently OutOfSync and can rotate the webhook Secret and caBundle during automated reconciliation.

Argo CD uses helm template to inflate charts rather than running a cluster-connected Helm install or upgrade. In templates/webhook.yaml, the chart calls genCA and genSignedCert, then attempts to preserve the live Secret with lookup. Because lookup cannot read the cluster during Argo CD rendering, a new CA, certificate, key, and webhook caBundle are desired on every comparison.

Reproduction

helm pull api7/api7-ingress-controller \
  --version 0.1.25 \
  --untar

for run in 1 2; do
  helm template gitops-test ./api7-ingress-controller \
    --namespace ingress-system \
    | sed -n '/kind: Secret/,/^---/p' \
    | sha256sum
done

The two hashes differ even though the chart, release name, namespace, and values are identical. The generated ValidatingWebhookConfiguration.webhooks[*].clientConfig.caBundle also differs.

Setting either of the following produces stable repeated renders:

• webhook.enabled=false
• webhook.certificate.provided=true with a stable caBundle

The external-certificate path is also fragile because the chart documents webhook.certificate.secretName, but the Deployment always mounts the helper-generated Secret name. The equivalent APISIX defect is tracked by apache/apisix-helm-chart#954 and PR apache/apisix-helm-chart#955.

The shared APISIX chart behavior is now tracked in apache/apisix-helm-chart#993. Argo CD's generic lack of cluster-backed Helm lookup support is tracked in argoproj/argo-cd#5202.

Expected behavior

Rendering the chart repeatedly with identical inputs should produce a stable desired state for GitOps reconcilers. Installing, refreshing, and synchronizing the chart through Argo CD should not rotate the webhook trust chain or report perpetual drift.

Possible designs

• Manage webhook certificate creation and CA injection at runtime with deterministic rendered manifests.
• Add a supported cert-manager/cainjector path.
• Complete and test the externally managed certificate path, including the configured Secret name.
• If temporary diff suppression is required, document the exact fields and its certificate-rotation implications rather than making it the chart default.

Acceptance criteria

• Two offline helm template renders with identical inputs produce identical webhook resources.
• An Argo CD application remains Synced after a hard refresh following the first successful sync.
• A second automated sync does not rotate the webhook Secret or CA bundle.
• Certificate renewal and chart upgrade behavior are covered by chart tests.
• The APISIX and API7 chart implementations remain aligned unless a product-specific difference is intentional and documented.

This currently blocks documenting the default chart path as a supported Argo CD installation. It does not block users who deliberately disable the webhook or supply and manage a stable certificate.

Start with templates/webhook.yaml and run the two-render helm template reproduction to confirm the changing Secret and caBundle. Trace the helper-generated Secret and configured secretName paths, then add or run the chart tests covering repeated offline renders, renewal, and upgrades. Done means identical webhook resources, no rotation on a second sync, and aligned APISIX/API7 behavior.