Wholistic approach for persisted config/metadata in Raft
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Nomad occasionally needs to persist some data a global metadata in Raft. For example, Nomad 0.12 persists autopilot config, scheduler config, and cluster id in Raft logs. Currently, Nomad attempts to create these on leadership establishment.
Auto-creating them on leadership creation has two downsides.
First, such creation performs a raft transaction with io/networking overhead, which slows down the server taking on leadership duties. This is a reasonable trade-off if the values are critical and needed for leadership duties, but for non-critical values, it's a small annoying overhead.
Second and more importantly, this approach doesn't handle upgrades well. Metadata creation method should only apply to raft if all servers have upgraded (to avoid emitting raft log entries un-interpretable by old servers) - but calling the method on leadership transition may mean that such metadata creation isn't guaranteed. Consider the case where a new server gains leadership with an old follower; when the old follower is upgraded, the creation function isn't invoked until the next leadership transition.
Consider adding another loop invoked on leadership transitions and when server compositions change to initialize and apply these metadata values.
Sample code for these methods: https://github.com/hashicorp/nomad/blob/v0.12.0/nomad/leader.go#L1411-L1436 .
A good workaround is to follow the cluster id pattern: The method [`ClusterID()`](https://github.com/hashicorp/nomad/blob/v0.12.0/nomad/server.go#L1639-L1676) returns the cluster id if one is already present, otherwise, it generates a new one, and is meant to be used anytime the cluster id is needed. On leadership election, the [method is called](https://github.com/hashicorp/nomad/blob/v0.12.0/nomad/leader.go#L240-L241), to initialize the value and speed up access. This approach works around the problem above, as leadership transition call is merely an optimization.
Though, we don't always follow the pattern: For scheduler/autopilot config, leadership is the only place where they are created and when it's needed, `state.SchedulerConfig()` is called directly.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the leadership-transition logic in nomad/leader.go and the ClusterID() implementation in nomad/server.go, then inspect the scheduler and autopilot configuration access paths, including state.SchedulerConfig(). Compare these with the sample methods from v0.12.0. Done should ensure persisted metadata is available without relying solely on leadership creation and remains safe during upgrades and server-composition changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100