node drain default behavior for -ignore-system
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
In https://github.com/hashicorp/nomad/pull/8606#discussion_r467480257 @jippi raised the question of whether `nomad node drain` should have the [`-ignore-system`](https://www.nomadproject.io/docs/commands/node/drain#ignore-system) flag set by default.
Currently the default behavior is that service and batch jobs are drained, and then system jobs are drained. However, internal allocation runner post-run hooks (ex. deregistering from Consul, cleaning up disk) don't block draining the system jobs, so they may run concurrently with the system jobs being drained. With the `-ignore-system` flag, the system jobs are never drained, which makes them available during shutdown.
The `-ignore-system` flag appears to be strictly more useful than not having it set. Some example scenarios where you'd want to have a system job that runs until all other workloads have completed:
* log shippers
* monitoring agents
* ingress proxies (ex. Nginx or HAProxy in front of web services)
* CSI node plugins
Changing the default behavior would break backwards compatibility, so we want to solicit feedback from the community about whether this change would be disruptive.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the `nomad node drain` command and its `-ignore-system` documentation, then review the linked pull request discussion and the documented current behavior. Done means reaching and recording a community-supported decision about changing the default, including the backward-compatibility impact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100