hashicorp / hashicorp/nomad

Nomad server leave automatically but not permanently

Open
#27,157 3 comments 0 reactions 0 assignees View on GitHub
hcc/jira stage/needs-discussion theme/autopilot type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal
Have Nomad servers (and potentially clients?) notify the leader whenever they interrupt/terminate. Something like [leave_on_interrupt](https://developer.hashicorp.com/nomad/docs/configuration#leave_on_interrupt) but then less permanent.

For follower-servers, we have to wait a while for the leader to pick up on the follower being offline.

Currently I'm not sure whether leader-servers already do this, but I'd imagine something like `nomad operator step-down` (doesn't exist in Nomad, but does in Vault) would be useful to have Nomad do on shutdown.

In cases where we *know* the server will be offline for a while (e.g. reboot), there's no need to wait for timeouts/heartbeats.

### Use-cases
- Checking server-health more reliably (e.g. when deciding "Can I, a follower node, reboot right now?`)

### Attempted Solutions
- Waiting about 5-10 seconds to see `Alive` change in `Left`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the existing leave_on_interrupt behavior and the Nomad server membership states mentioned in the issue, including Alive and Left. Compare the requested behavior with the nomad operator step-down precedent, then clarify whether the scope includes follower servers, leader servers, or clients; done requires an agreed shutdown-notification design that avoids waiting for timeout or heartbeat detection.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.