apache / apache/apisix-helm-chart
[apisix-ingress-controller] webhook certificate rendering causes perpetual Argo CD drift
- Dominant language
- Go Template
- Stars
- 289
- Forks
- 282
- Avg merge
- 15h 55m
- Merged PRs (30d)
- 3
Description
### Affected chart
- `apisix/apisix-ingress-controller` 1.2.2
- Current `master` has the same template behavior
### Problem
The Ingress Controller chart produces different webhook certificate data every time it is rendered without cluster access. This leaves an Argo CD application permanently `OutOfSync` and can rotate the webhook Secret and `caBundle` during automated reconciliation.
Argo CD uses `helm template` to inflate charts rather than running a cluster-connected Helm install or upgrade. In `templates/webhook.yaml`, the chart calls `genCA` and `genSignedCert`, then attempts to preserve the live Secret with `lookup`. Because `lookup` cannot read the cluster during Argo CD rendering, a new CA, certificate, key, and webhook `caBundle` are desired on every comparison.
### Reproduction
```shell
helm pull apisix/apisix-ingress-controller \
--version 1.2.2 \
--untar
for run in 1 2; do
helm template gitops-test ./apisix-ingress-controller \
--namespace ingress-system \
| sed -n '/kind: Secret/,/^---/p' \
| sha256sum
done
```
The two hashes differ even though the chart, release name, namespace, and values are identical. The generated `ValidatingWebhookConfiguration.webhooks[*].clientConfig.caBundle` also differs.
Setting either of the following produces stable repeated renders:
- `webhook.enabled=false`
- `webhook.certificate.provided=true` with a stable `caBundle`
The external-certificate path is currently incomplete because `webhook.certificate.secretName` is ignored. That separate defect is tracked by #954 and PR #955. Merging #955 makes a workaround less fragile, but does not make the default chart render deterministic.
Argo CD's generic lack of cluster-backed Helm `lookup` support is tracked in argoproj/argo-cd#5202. The chart should not require that behavior to reach a stable desired state.
### Expected behavior
Rendering the chart repeatedly with identical inputs should produce a stable desired state for GitOps reconcilers. Installing, refreshing, and synchronizing the chart through Argo CD should not rotate the webhook trust chain or report perpetual drift.
### Possible designs
- Manage webhook certificate creation and CA injection at runtime with deterministic rendered manifests.
- Add a supported cert-manager/cainjector path.
- Complete and test the externally managed certificate path, including the configured Secret name.
- If temporary diff suppression is required, document the exact fields and its certificate-rotation implications rather than making it the chart default.
### Acceptance criteria
- Two offline `helm template` renders with identical inputs produce identical webhook resources.
- An Argo CD application remains `Synced` after a hard refresh following the first successful sync.
- A second automated sync does not rotate the webhook Secret or CA bundle.
- Certificate renewal and chart upgrade behavior are covered by chart tests.
This currently blocks documenting the default chart path as a supported Argo CD installation. It does not block users who deliberately disable the webhook or supply and manage a stable certificate.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with templates/webhook.yaml and reproduce the differing output using the provided offline helm template commands. Compare the Secret and ValidatingWebhookConfiguration caBundle across repeated renders, then review the chart tests and the certificate paths described in the issue. Done means identical offline renders, no repeated Argo CD drift or certificate rotation, and coverage for renewal and chart upgrades.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100