Dstack-TEE / Dstack-TEE/dstack

gateway: concurrent ACME rotation is only best-effort serialized

Offen
#1,008 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Rust
Sterne
544
Forks
96
Ø Merge
17 Std. 57 Min.
Gemergte PRs (30 T.)
117

Beschreibung

Follow-up to #935.

`RotateAcmeCredentials` is serialized across nodes by a TTL lock in WaveKV (`global/acme_rotation_lock`). WaveKV is last-writer-wins without compare-and-swap, so the lock is acquired by read-then-write: two nodes calling the RPC within a replication gap can both acquire it and rotate concurrently.

## Impact

If two rotations interleave, CAA records end up pinned to one node's new account while LWW keeps the other node's credential in KV. Since #935 the state is recoverable — the published credential wins LWW and one `SetCaa` run re-pins every domain to it — but issuance is broken until an operator notices and intervenes.

The constraint "rotate through one gateway at a time" is advisory only: the admin endpoint uses a shared bearer token with no per-method authorization, so nothing enforces it.

## Possible directions

- Add CAS (or a fenced-lock primitive) to WaveKV and make the rotation lock a real mutex.
- Route rotation to a designated leader node instead of accepting it on any gateway.
- At minimum: automatic post-rotation verification that the published credential's `accounturi` matches every domain's CAA, alerting on divergence (see the CAA reconciliation issue).

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start by reading the RotateAcmeCredentials RPC, the WaveKV global/acme_rotation_lock behavior, and the follow-up context in #935. Compare the listed CAS or fenced-lock, leader-routing, and post-rotation verification directions with the gateway’s current behavior. Done should prevent concurrent rotations or reliably detect and recover from CAA and credential divergence.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
rust
Bereich
distributed-systems
Issue-Typ
Bug
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Ruhig
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.