hashicorp / hashicorp/nomad

Node reboot manager

Open
#15,716 5 comments 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/autopilot theme/autoscaling type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal
Something similar to what's provided in Kubernetes by tools like [Kured](https://github.com/kubereboot/kured), [locksmith](https://github.com/flatcar/locksmith/) and [Airlock](https://github.com/coreos/airlock)

To assist with performing a safe rolling restart/redeploy of servers running in a Nomad cluster, commonly after updating the host OS, Nomad could perform this reboot itself but only after it has ensured that this will not break quorum (for server nodes) and not before properly draining a node of it's tasks.

If this could additionally take into account Consul and Vault this would be even better and provides for a very nicely integrated solution, I assume a lot of people run Consul/Vault servers on the same nodes that also run Nomad servers; or if they are properly separated a Nomad client might still be present as well.

Kured can be triggered using either a command or by watching for the existence of the `/var/run/reboot-required` file, would Nomad also check for that file it'll gain out of the box integration with Flatcar (and possibly others that follow suit) for free.
For Fedora CoreOS Zincati is used, it's implementation differs in that it works with the FleetLock protocol over HTTP but the idea is the same.

### Use-cases
Automated node reboots, especially on a container OS like Flatcar that provides automatic updates and a post-update trigger mechanism for the reboot.

### Attempted Solutions
Use locksmith or Airlock which also requires you to run etcd which doesn't make a lot of sense when already running Consul, combined with some wrapper tool that uses the Nomad API to drain a node beforehand.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the Nomad API behavior for draining a node, then examine how server quorum and client task placement are represented. Done would be an integrated reboot workflow that safely drains tasks, preserves server quorum, and supports the reboot triggers described, including /var/run/reboot-required and FleetLock.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.