hcengineering / hcengineering/huly-selfhost
Helm: chart-generated secrets are regenerated on every ArgoCD sync (lookup is empty without cluster access)
- Dominant language
- Shell
- Stars
- 3.5k
- Forks
- 469
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
`helm/huly/templates/secret.yaml` keeps generated secrets stable across upgrades by looking up the
Secret already in the cluster:
1. explicit `.Values.secrets.*`, else
2. `lookup "v1" "Secret" .Release.Namespace $secretName`, else
3. `randAlphaNum`
`lookup` only returns data when Helm has a live cluster connection. `helm template` — which is what
ArgoCD (and Flux, and `--dry-run`) renders with — never performs it, so branch 2 is always empty and
every render falls through to `randAlphaNum`.
The result is that each sync mints a fresh `SERVER_SECRET`, `COCKROACH_PASSWORD` and
`REDPANDA_SUPERUSER_PASSWORD`. A new `SERVER_SECRET` invalidates every issued JWT, so all users are
signed out on every sync, and the rotated database passwords no longer match the data already
initialised in CockroachDB.
## Reproduction
No cluster needed — render the chart twice:
```console
$ helm template huly ./helm/huly --set domain=example.com | grep '^ SERVER_SECRET'
SERVER_SECRET: "Q3RPRFl6M2JCMnBLZ3ozeEdYZFRYR0RrMHBuazJ4TGg="
$ helm template huly ./helm/huly --set domain=example.com | grep '^ SERVER_SECRET'
SERVER_SECRET: "RlFHdUlTZWhrQ0lqdjZNZDVrTWVRS1RBRktuYnNHeXc="
```
`STORAGE_CONFIG` and `CR_DB_URL` change the same way. Helm 3.20.0, chart at `8655845`.
Under ArgoCD this manifest is what gets applied, so the Secret is rewritten on every sync and the
services restart with new values (`huly.checksumAnnotations` rolls the pods).
## Current workarounds
Both are unpleasant:
- put the real secrets in `values.yaml` — plaintext credentials in git, which is exactly what a
GitOps repo should not hold;
- add an ArgoCD `ignoreDifferences` entry for the Secret — the manifest still churns, and the
divergence between desired and live state has to be maintained by hand.
## Proposed fix
Support `secrets.existingSecret`, the convention used by Bitnami and most charts: when set, the
chart renders no Secret of its own and points every service at the named one. That makes the chart
usable with External Secrets Operator, Sealed Secrets, Vault, or a plain `kubectl create secret`,
and leaves the default auto-generating behaviour untouched.
## Unrelated observation from the same file
With `cockroach.enabled=false` and `secrets.crDbUrl` unset, the chart still renders
```
postgres://selfhost:@cockroach:26257/defaultdb
```
pointing at a Service that deployment never creates. `values.yaml` does document that `crDbUrl`
must be set for an external database, so this is a missing guard rather than a wrong default — but
failing at template time (as the chart already does for `secrets.openaiApiKey` when
`aibot.enabled=true`) would turn a confusing runtime connection error into a clear message. Happy
to send that separately if it is wanted.
## Environment
ArgoCD-managed Kubernetes, chart from `main` (`8655845`), Helm 3.20.0.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with helm/huly/templates/secret.yaml and values.yaml, then render the chart twice with the documented helm template command to observe the changing secrets. Trace how the generated Secret is referenced by services. Done means an optional secrets.existingSecret causes the chart to render no generated Secret and use the named Secret, while default auto-generation remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100