hashicorp / hashicorp/nomad

Documentation needed for signal handling/hot-restarts

Open
#6,700 4 comments 0 reactions 0 assignees View on GitHub
theme/docs
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

As an operator of Nomad since version 0.5.4, I was completely unaware of Nomad's ability to restart clients without impacting running allocations. I was only able to piece together this information by reading various snippets from PRs and issues.

For nearly the last ~3 years, I've been using shutdown scripts with locking mechanisms that drain nomad clients before a full shutdown (or restart). To provide context as to why I care about restarts; in most cases, we were restarting frequently to reload of TLS configurations for nomad and vault.

Back when I initially adopted Nomad, there were no best practices around signal handling or example systemd unit files. While there are now example systemd unit files, there is no corresponding documentation to explain how nomad servers/clients handle signals nor any documentation about nomad client startup logic. Improving/creating this documentation would be incredibly useful to operators; the lack of this has caused me considerable headaches and over-engineering over the years, and my experience with Nomad would have been significantly more positive had I known about this from the beginning. Right now I think operators are mostly left to their imagination to determine best practices. It's possible that it is documented somewhere, so apologies if it does exist and I've just missed it!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the existing example systemd unit files and the documentation covering Nomad client and server operation. Document how signals, client startup, hot restarts, and TLS-related restarts are handled, including practical operator guidance and best practices. Done means operators can understand the behavior without piecing it together from PRs and issues.

Written by the indexing model from the issue text.

Assessment

Domain
devops, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.