Render-time generated credentials drift under GitOps: fan out existing-Secret options and provision them from a pre-install hook

Problem

The wrapped upstream Harbor chart mints several credentials at render time when they are not supplied: the core component secret and CSRF key (randAlphaNum), the token-signing key pair (genCA), the jobservice and registry HTTP secrets (randAlphaNum), and the registry htpasswd entry (htpasswd, salted). Upstream relies on lookup to reuse what already exists in the cluster.

Under a GitOps controller that renders without cluster access (Argo CD, Flux's helm-controller in some modes), lookup returns nothing, every render draws fresh values, and with self-heal enabled each sync re-keys four Secrets and rolls core, jobservice, and registry. This is Argo CD's documented "random data" failure mode. Reproduced against this chart at 877dbab: two helm template runs with identical values differ in exactly seven objects.

Consumers can work around it today by creating two Secrets by hand before the first sync and pointing upstream's existing-Secret options at them (verified: two renders become byte-identical):

• an Opaque Secret with secretKey (16 chars), secret, CSRF_KEY, JOBSERVICE_SECRET, REGISTRY_HTTP_SECRET, REGISTRY_PASSWD, REGISTRY_HTPASSWD → existingSecretSecretKey, core.existingSecret, core.existingXsrfSecret, jobservice.existingSecret, registry.existingSecret, registry.credentials.existingSecret
• a kubernetes.io/tls Secret → core.secretName

That is seven values keys and two hand-built Secrets on every cluster, and the htpasswd entry needs a bcrypt tool the operator may not have. It belongs in the pack.

Proposal

Two parts, either of which helps on its own:

1. A stableSecrets values block that fans out to all seven upstream keys from two Secret names:

   stableSecrets:
     enabled: true
     internalSecret: harbor-internal        # Opaque, keys as above
     tokenSigningSecret: harbor-token-signing  # kubernetes.io/tls
   

   with the chart's own values.yaml documenting exactly which keys each Secret must carry. This removes the need for consumers to know upstream's option names.

2. Provision the Secrets from the bootstrap path (#2/#3 add a post-install job; this wants a pre-install/PreSync hook, since the pods reference the Secrets at start): if the named Secret does not exist, generate the material (openssl for the key pair; the bcrypt entry can be produced with htpasswd from an httpd image or a small Go/Python helper) and create it; if it exists, leave it untouched. Never log values. The admin password (harbor-admin) could be generated the same way when absent, which would remove the last manual pre-sync step.

Acceptance

• Two helm template runs with identical values produce byte-identical output when stableSecrets is configured (add this as a CI check; it is the regression that matters).
• With provisioning enabled, a fresh install on a cluster with no pre-created Secrets comes up with stable credentials, and a second sync changes no Secret data and rolls no pod.
• Rotating a credential is documented: recreate the Secret, restart the mounting pods.
• README states which Secrets exist, what they protect (notably that secretKey encrypts stored robot/registry credentials, so it must be backed up with the data).

Context

Found deploying the pack through Argo CD for the Collab Hub cog-registry work (nebari-dev/collab-hub-pack#7). Related: #2 (declarative projects), #3 (robot Secrets), #6 (in-cluster consumer docs; the token-signing Secret and secretKey backup note belong there too).

Start with values.yaml and the chart's bootstrap path, then review the existing upstream Secret options described in the issue. Use two identical helm template runs as the determinism check, and verify the provisioning behavior against the acceptance criteria. Update README with the Secret contents, protections, and rotation procedure.