linode / linode/apl-core

argocd-redis password split: redis never restarts on Secret rotation → all Applications ComparisonError (WRONGPASS)

Open
#3,424 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

community v6.3.0
Dominant language
Go Template
Stars
2.3k
Forks
186
Avg merge
3d 11h
Merged PRs (30d)
66

Description

## Summary

`argocd-redis` and its clients (repo-server, application-controller) both derive the Redis password from the `argocd-redis` Secret, but each reads it **once at container start**. When that Secret's value is rewritten while the `argocd-redis` pod keeps running, the server and clients diverge and **every** Argo CD Application flips to `Unknown/ComparisonError`:

```
ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 1:
rpc error: code = Unknown desc = failed to list refs:
WRONGPASS invalid username-password pair or user is disabled.
```

The repo-server caches git refs/manifests in `argocd-redis`, so a bad AUTH turns *all* manifest generation into a ComparisonError at once — the whole app-of-apps stalls.

## Mechanism

- `argocd-redis` Deployment runs `redis-server --requirepass $(REDIS_PASSWORD)`, with `REDIS_PASSWORD` sourced from Secret `argocd-redis` key `auth`. The value is substituted into `--requirepass` **at pod start** and never re-read.
- `argocd-repo-server` / `argocd-application-controller` read `REDIS_PASSWORD` from the **same** Secret key, also at their pod start.
- If the `argocd-redis` Secret is rewritten (e.g. the ExternalSecret backing `auth` rotates, or the secret-init regenerates it) **without restarting the redis pod**, then:
- redis keeps requiring the **old** password;
- any client pod that (re)starts after the rewrite reads the **new** password;
- every client→redis `AUTH` fails with `WRONGPASS` → all Applications ComparisonError.

Nothing restarts `argocd-redis` when its password Secret changes, so server and clients can silently diverge.

## Where we hit it

A reused (kept) e2e cluster. Observed state at failure:

| Pod | Age | Restarts | Password source |
|---|---|---|---|
| `argocd-redis` | 121m | **0** | `argocd-redis/auth` (baked into `--requirepass` at 17:16) |
| `argocd-repo-server` ×3 | 5–7m | 0 | `argocd-redis/auth` (read at 19:11–19:13) |
| `argocd-application-controller-0` | ~7m | 0 | `argocd-redis/auth` (read ~19:11) |

The redis pod outlived a rewrite of the `argocd-redis` Secret; the repo-servers rolled and picked up the new value; redis (never restarted) kept the old one → WRONGPASS across all 55 Applications. It also reproduces any time the clients roll (e.g. HPA scale-up, a chart re-sync) after the Secret changes under a long-lived redis pod.

## Impact

- Every Argo CD Application → `Unknown/ComparisonError`; the app-of-apps cannot reconcile until redis is manually restarted.
- Cosmetically indistinguishable from a mass config failure, so it's easy to misdiagnose.
- Workaround: `kubectl -n argocd rollout restart deploy/argocd-redis` (redis re-reads the current Secret and realigns with the clients).

## Requested fix (either is sufficient)

1. **Restart-on-rotation** — add a `checksum/secret` (of the `argocd-redis` `auth`) annotation to the `argocd-redis` pod template so a password change rolls redis in lockstep with the clients that consume it. This is the standard Helm pattern and closes the divergence window.
2. **Static password** — make the `argocd-redis` password stable (seed once, never rotate), so client rolls can never diverge from a never-restarted redis.

Either eliminates the split. (1) is the more general fix since it also covers deliberate rotations.

## Environment

- Argo CD `v3.4.3`, `argocd-redis` image `redis:8.2.3-alpine`, redis auth via `argocd-redis` Secret (`redis-username` / `auth`), populated by an ExternalSecret.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the Helm templates for the argocd-redis Deployment, pod template, and argocd-redis Secret, then trace how the auth value is consumed. Verify the chosen fix causes redis to restart when the auth value changes and that the resulting server and clients use the same password; reproduce or validate the rotation scenario with the repository's Kubernetes checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, redis
Domain
databases, devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.