Dstack-TEE / Dstack-TEE/dstack
gateway: concurrent ACME rotation is only best-effort serialized
- Vorherrschende Sprache
- Rust
- Sterne
- 544
- Forks
- 96
- Ø Merge
- 17 Std. 57 Min.
- Gemergte PRs (30 T.)
- 117
Beschreibung
Follow-up to #935.
`RotateAcmeCredentials` is serialized across nodes by a TTL lock in WaveKV (`global/acme_rotation_lock`). WaveKV is last-writer-wins without compare-and-swap, so the lock is acquired by read-then-write: two nodes calling the RPC within a replication gap can both acquire it and rotate concurrently.
## Impact
If two rotations interleave, CAA records end up pinned to one node's new account while LWW keeps the other node's credential in KV. Since #935 the state is recoverable — the published credential wins LWW and one `SetCaa` run re-pins every domain to it — but issuance is broken until an operator notices and intervenes.
The constraint "rotate through one gateway at a time" is advisory only: the admin endpoint uses a shared bearer token with no per-method authorization, so nothing enforces it.
## Possible directions
- Add CAS (or a fenced-lock primitive) to WaveKV and make the rotation lock a real mutex.
- Route rotation to a designated leader node instead of accepting it on any gateway.
- At minimum: automatic post-rotation verification that the published credential's `accounturi` matches every domain's CAA, alerting on divergence (see the CAA reconciliation issue).
Beitragsleitfaden
Rechercherichtung
Start by reading the RotateAcmeCredentials RPC, the WaveKV global/acme_rotation_lock behavior, and the follow-up context in #935. Compare the listed CAS or fenced-lock, leader-routing, and post-rotation verification directions with the gateway’s current behavior. Done should prevent concurrent rotations or reliably detect and recover from CAA and credential divergence.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- rust
- Bereich
- distributed-systems
- Issue-Typ
- Bug
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Ruhig
- Klarheit
- Muss geklärt werden
- Anfängerfreundlichkeit
- 35/100