Nomad-Consul sync for background SI token reconciliation
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
As part of #6701 Nomad Servers will need to be able to perform background Service Identity token reconciliation.
There is an inherent race condition where a Nomad Server successfully requests a new SI token from Consul, but then crashes / loses leadership before it's able to persist the details of that token into the raft log. This isn't particularly bad for Nomad, which can simply retry the request later on with a new leader, but it could cause orphaned tokens to accumulate in Consul. Since SI tokens are not periodic and will have no TTL, such tokens would linger forever. To avoid that, Nomad Servers can periodically request from Consul a list of every token it knows about, and compare that with every SI token a Nomad cluster knows about. If there are any SI tokens in Consul generated for the Nomad cluster (compared with metadata stored in the Description field), that Nomad is not aware of, request Consul to revoke them.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with issue #6701 and the Nomad server work for Service Identity tokens. Trace how token metadata is persisted in the raft log and how Consul lists and revokes tokens; done means background reconciliation removes Consul tokens for the Nomad cluster that Nomad does not know about.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100