hashicorp / hashicorp/nomad

document/improve on client restarts with missing state

Open
#9,512 0 comments 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/docs
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When a Nomad client is stopped, the allocations on that client host are left running. So long as the client isn't offline long enough to be considered "lost", when the client restarts it rummages around in its local state store to recreate handles to the running tasks. If a task is stopped while the Nomad client is stopped (by the user or simply crashing), the Nomad client has to restore the task. Any failure to do so is definitely a Nomad bug.

However, we've seen operators who remove the client data directory between restarts. There are two ways we've seen this go wrong:

* If the client's data directory is removed while the client is shut down, the Nomad client has no way of recreating the handles to running tasks. This also means that Nomad can't shut down or restart those tasks, which could result in stale versions of applications can be running.
* If the client's data directory is removed and the task containers are removed manually, but some other resource like an un-garbage-collected mount is left behind, this can prevent Nomad from scheduling the workload.

Many operators (typically those who are running on public cloud infra) will replace the client host entirely during client upgrades. But for those who do not, generally speaking they should not remove the data dir on the client. If they do they need to be aware of all the resources that can be leaked. We don't have good documentation warning about this or giving guidance on it.

Contributor guide

No contributing guide indexed for this repository

Research direction

No documentation file is named. Start by searching the Nomad documentation for client data directories, restarts, and lost allocations. Done means adding guidance that explains both data-directory removal scenarios, the resulting leaked resources, and the recommendation to preserve the directory or replace the host.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.