hcengineering / hcengineering/huly-selfhost

Helm: chart-generated secrets are regenerated on every ArgoCD sync (lookup is empty without cluster access)

Open
#321 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
3.5k
Forks
469
PR merge metrics
No merged PRs in 30d

Description

## Summary

`helm/huly/templates/secret.yaml` keeps generated secrets stable across upgrades by looking up the
Secret already in the cluster:

1. explicit `.Values.secrets.*`, else
2. `lookup "v1" "Secret" .Release.Namespace $secretName`, else
3. `randAlphaNum`

`lookup` only returns data when Helm has a live cluster connection. `helm template` — which is what
ArgoCD (and Flux, and `--dry-run`) renders with — never performs it, so branch 2 is always empty and
every render falls through to `randAlphaNum`.

The result is that each sync mints a fresh `SERVER_SECRET`, `COCKROACH_PASSWORD` and
`REDPANDA_SUPERUSER_PASSWORD`. A new `SERVER_SECRET` invalidates every issued JWT, so all users are
signed out on every sync, and the rotated database passwords no longer match the data already
initialised in CockroachDB.

## Reproduction

No cluster needed — render the chart twice:

```console
$ helm template huly ./helm/huly --set domain=example.com | grep '^ SERVER_SECRET'
SERVER_SECRET: "Q3RPRFl6M2JCMnBLZ3ozeEdYZFRYR0RrMHBuazJ4TGg="
$ helm template huly ./helm/huly --set domain=example.com | grep '^ SERVER_SECRET'
SERVER_SECRET: "RlFHdUlTZWhrQ0lqdjZNZDVrTWVRS1RBRktuYnNHeXc="
```

`STORAGE_CONFIG` and `CR_DB_URL` change the same way. Helm 3.20.0, chart at `8655845`.

Under ArgoCD this manifest is what gets applied, so the Secret is rewritten on every sync and the
services restart with new values (`huly.checksumAnnotations` rolls the pods).

## Current workarounds

Both are unpleasant:

- put the real secrets in `values.yaml` — plaintext credentials in git, which is exactly what a
GitOps repo should not hold;
- add an ArgoCD `ignoreDifferences` entry for the Secret — the manifest still churns, and the
divergence between desired and live state has to be maintained by hand.

## Proposed fix

Support `secrets.existingSecret`, the convention used by Bitnami and most charts: when set, the
chart renders no Secret of its own and points every service at the named one. That makes the chart
usable with External Secrets Operator, Sealed Secrets, Vault, or a plain `kubectl create secret`,
and leaves the default auto-generating behaviour untouched.

## Unrelated observation from the same file

With `cockroach.enabled=false` and `secrets.crDbUrl` unset, the chart still renders

```
postgres://selfhost:@cockroach:26257/defaultdb
```

pointing at a Service that deployment never creates. `values.yaml` does document that `crDbUrl`
must be set for an external database, so this is a missing guard rather than a wrong default — but
failing at template time (as the chart already does for `secrets.openaiApiKey` when
`aibot.enabled=true`) would turn a confusing runtime connection error into a clear message. Happy
to send that separately if it is wanted.

## Environment

ArgoCD-managed Kubernetes, chart from `main` (`8655845`), Helm 3.20.0.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with helm/huly/templates/secret.yaml and values.yaml, then render the chart twice with the documented helm template command to observe the changing secrets. Trace how the generated Secret is referenced by services. Done means an optional secrets.existingSecret causes the chart to render no generated Secret and use the named Secret, while default auto-generation remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.