Node reboot manager
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Proposal
Something similar to what's provided in Kubernetes by tools like [Kured](https://github.com/kubereboot/kured), [locksmith](https://github.com/flatcar/locksmith/) and [Airlock](https://github.com/coreos/airlock)
To assist with performing a safe rolling restart/redeploy of servers running in a Nomad cluster, commonly after updating the host OS, Nomad could perform this reboot itself but only after it has ensured that this will not break quorum (for server nodes) and not before properly draining a node of it's tasks.
If this could additionally take into account Consul and Vault this would be even better and provides for a very nicely integrated solution, I assume a lot of people run Consul/Vault servers on the same nodes that also run Nomad servers; or if they are properly separated a Nomad client might still be present as well.
Kured can be triggered using either a command or by watching for the existence of the `/var/run/reboot-required` file, would Nomad also check for that file it'll gain out of the box integration with Flatcar (and possibly others that follow suit) for free.
For Fedora CoreOS Zincati is used, it's implementation differs in that it works with the FleetLock protocol over HTTP but the idea is the same.
### Use-cases
Automated node reboots, especially on a container OS like Flatcar that provides automatic updates and a post-update trigger mechanism for the reboot.
### Attempted Solutions
Use locksmith or Airlock which also requires you to run etcd which doesn't make a lot of sense when already running Consul, combined with some wrapper tool that uses the Nomad API to drain a node beforehand.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the Nomad API behavior for draining a node, then examine how server quorum and client task placement are represented. Done would be an integrated reboot workflow that safely drains tasks, preserves server quorum, and supports the reboot triggers described, including /var/run/reboot-required and FleetLock.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100