linode / linode/apl-core

PRD: Run Dex in HA and reload it on configuration change

Open
#3,539 0 comments 0 reactions 0 assignees View on GitHub
not-ready ready-for-agent
Dominant language
Go Template
Stars
2.3k
Forks
186
Avg merge
3d 11h
Merged PRs (30d)
66

Description

## Problem Statement

Dex reads its configuration only at startup. There is no reload, no file watch and no signal handler for it; only TLS certificates are reloaded at runtime.

Users no longer live in that configuration — #3541 provisions them over Dex's gRPC API instead, so a user or team-membership change is effective immediately with no restart. OAuth2 clients still do: APL holds one client per team, and ordinary platform activity changes that. Adding a team adds hostnames, which adds redirect URIs to a client's configuration. That change reaches the configuration secret and then does nothing, because the running process never re-reads it. The change appears to have been applied while having no effect, and users of that team's applications cannot complete a redirect.

Dex is also a single point of failure. Every login goes through it, and oauth2-proxy will not start until it answers, so one pod restarting or being evicted takes platform authentication with it.

## Solution

Run Dex with more than one replica, and roll it automatically when its configuration changes.

Replicas are safe because Dex keeps nothing in pod memory that matters. Sessions, authorization codes, refresh tokens and signing keys all live in its storage, and the session cookie encryption key is generated once and shared by every replica. That last point is what makes replicas interchangeable rather than merely present; without a shared key a session created by one pod is unreadable by another.

A checksum of the rendered configuration is annotated on the deployment. When the configuration changes the annotation changes and Kubernetes rolls the pods. Combined with a rollout that starts a new pod before stopping an old one, the restart is invisible and nobody is logged out.

## Acceptance Criteria

1. **GIVEN** default values, **WHEN** the platform is rendered, **THEN** Dex runs more than one replica.
2. **GIVEN** more than one replica, **WHEN** the platform is rendered, **THEN** a disruption budget keeps at least one pod available and the pods are spread across nodes without preventing scheduling on a single-node cluster.
3. **GIVEN** a configuration change, **WHEN** the platform is rendered, **THEN** the deployment's configuration checksum changes.
4. **GIVEN** no configuration change, **WHEN** the platform is rendered repeatedly, **THEN** the checksum is stable, so no pod is rolled without cause.
5. **GIVEN** a new team is added, **WHEN** the platform is applied, **THEN** the checksum changes, Dex restarts, and users of that team's applications can authenticate.
6. **GIVEN** a configuration change on a running cluster, **WHEN** the rollout proceeds, **THEN** at least one pod serves throughout and no authenticated user is logged out.
7. **GIVEN** the rendered configuration, **WHEN** it is compared with the text that is hashed, **THEN** they are produced from the same source, so the checksum cannot drift from what is deployed.

## Testing

- Render the platform twice unchanged and confirm the checksum is identical; change a team and confirm it differs and then returns when reverted. This can be done entirely from templates.
- Criteria 5 and 6 need a cluster: apply a change, watch the rollout, and confirm an existing session survives it.
- Worth verifying that two replicas can each serve a session created by the other, since that is the property the shared encryption key exists to provide.

## Out of Scope

- Autoscaling Dex.
- Reload-on-change without a restart. Client/redirect-URI changes still need one; only user changes were removed from this path, by #3541.
- Backing up Dex's storage.

## Further Notes

- The checksum covers the rendered configuration, in which secret values are still template expressions rather than resolved values. A rotated secret therefore does not roll the pods. Acceptable, because rotation is rare and deliberate, but it means a rotation needs a manual restart and should be documented as such.
- Restarting is cheap here specifically because sessions are stored rather than held in memory. Since #3541, this restart only fires on team/client config changes (new hostnames, redirect URIs) — not on individual user creation, update or deletion, which no longer touch configuration at all.

## Dependencies

Depends on #3536.

Not required by the issuer switch. Since #3541 moved user provisioning off the config file, the checksum here only needs to cover team/OAuth2-client changes, so this can land independently of #3541 rather than close behind it.

Contributor guide

Open the contributing guide

Research direction

Start with the platform's Kubernetes templates and the existing Dex deployment and configuration rendering; the issue does not name specific files or tests. Render the platform repeatedly to verify stable and changed checksums, then use a cluster rollout to verify availability, session continuity, and cross-replica session handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.