hashicorp / hashicorp/nomad

Nomad-Consul sync for background SI token reconciliation

Open
#6,719 1 comment 0 reactions 0 assignees View on GitHub
theme/consul/connect type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

As part of #6701 Nomad Servers will need to be able to perform background Service Identity token reconciliation.

There is an inherent race condition where a Nomad Server successfully requests a new SI token from Consul, but then crashes / loses leadership before it's able to persist the details of that token into the raft log. This isn't particularly bad for Nomad, which can simply retry the request later on with a new leader, but it could cause orphaned tokens to accumulate in Consul. Since SI tokens are not periodic and will have no TTL, such tokens would linger forever. To avoid that, Nomad Servers can periodically request from Consul a list of every token it knows about, and compare that with every SI token a Nomad cluster knows about. If there are any SI tokens in Consul generated for the Nomad cluster (compared with metadata stored in the Description field), that Nomad is not aware of, request Consul to revoke them.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with issue #6701 and the Nomad server work for Service Identity tokens. Trace how token metadata is persisted in the raft log and how Consul lists and revokes tokens; done means background reconciliation removes Consul tokens for the Nomad cluster that Nomad does not know about.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.