Dstack-TEE / Dstack-TEE/dstack
gateway: concurrent ACME rotation is only best-effort serialized
- Dominant language
- Rust
- Stars
- 544
- Forks
- 96
- Avg merge
- 17h 57m
- Merged PRs (30d)
- 117
Description
Follow-up to #935.
`RotateAcmeCredentials` is serialized across nodes by a TTL lock in WaveKV (`global/acme_rotation_lock`). WaveKV is last-writer-wins without compare-and-swap, so the lock is acquired by read-then-write: two nodes calling the RPC within a replication gap can both acquire it and rotate concurrently.
## Impact
If two rotations interleave, CAA records end up pinned to one node's new account while LWW keeps the other node's credential in KV. Since #935 the state is recoverable — the published credential wins LWW and one `SetCaa` run re-pins every domain to it — but issuance is broken until an operator notices and intervenes.
The constraint "rotate through one gateway at a time" is advisory only: the admin endpoint uses a shared bearer token with no per-method authorization, so nothing enforces it.
## Possible directions
- Add CAS (or a fenced-lock primitive) to WaveKV and make the rotation lock a real mutex.
- Route rotation to a designated leader node instead of accepting it on any gateway.
- At minimum: automatic post-rotation verification that the published credential's `accounturi` matches every domain's CAA, alerting on divergence (see the CAA reconciliation issue).
Contributor guide
Research direction
Start by reading the RotateAcmeCredentials RPC, the WaveKV global/acme_rotation_lock behavior, and the follow-up context in #935. Compare the listed CAS or fenced-lock, leader-routing, and post-rotation verification directions with the gateway’s current behavior. Done should prevent concurrent rotations or reliably detect and recover from CAA and credential divergence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100