hashicorp / hashicorp/nomad

Wholistic approach for persisted config/metadata in Raft

Open
#8,472 0 comments 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/raft
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Nomad occasionally needs to persist some data a global metadata in Raft. For example, Nomad 0.12 persists autopilot config, scheduler config, and cluster id in Raft logs. Currently, Nomad attempts to create these on leadership establishment.

Auto-creating them on leadership creation has two downsides.

First, such creation performs a raft transaction with io/networking overhead, which slows down the server taking on leadership duties. This is a reasonable trade-off if the values are critical and needed for leadership duties, but for non-critical values, it's a small annoying overhead.

Second and more importantly, this approach doesn't handle upgrades well. Metadata creation method should only apply to raft if all servers have upgraded (to avoid emitting raft log entries un-interpretable by old servers) - but calling the method on leadership transition may mean that such metadata creation isn't guaranteed. Consider the case where a new server gains leadership with an old follower; when the old follower is upgraded, the creation function isn't invoked until the next leadership transition.

Consider adding another loop invoked on leadership transitions and when server compositions change to initialize and apply these metadata values.

Sample code for these methods: https://github.com/hashicorp/nomad/blob/v0.12.0/nomad/leader.go#L1411-L1436 .

A good workaround is to follow the cluster id pattern: The method [`ClusterID()`](https://github.com/hashicorp/nomad/blob/v0.12.0/nomad/server.go#L1639-L1676) returns the cluster id if one is already present, otherwise, it generates a new one, and is meant to be used anytime the cluster id is needed. On leadership election, the [method is called](https://github.com/hashicorp/nomad/blob/v0.12.0/nomad/leader.go#L240-L241), to initialize the value and speed up access. This approach works around the problem above, as leadership transition call is merely an optimization.

Though, we don't always follow the pattern: For scheduler/autopilot config, leadership is the only place where they are created and when it's needed, `state.SchedulerConfig()` is called directly.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the leadership-transition logic in nomad/leader.go and the ClusterID() implementation in nomad/server.go, then inspect the scheduler and autopilot configuration access paths, including state.SchedulerConfig(). Compare these with the sample methods from v0.12.0. Done should ensure persisted metadata is available without relying solely on leadership creation and remains safe during upgrades and server-composition changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.