hashicorp / hashicorp/nomad

node drain default behavior for -ignore-system

Open
#8,622 1 comment 1 reaction 0 assignees View on GitHub
stage/needs-discussion theme/drain type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

In https://github.com/hashicorp/nomad/pull/8606#discussion_r467480257 @jippi raised the question of whether `nomad node drain` should have the [`-ignore-system`](https://www.nomadproject.io/docs/commands/node/drain#ignore-system) flag set by default.

Currently the default behavior is that service and batch jobs are drained, and then system jobs are drained. However, internal allocation runner post-run hooks (ex. deregistering from Consul, cleaning up disk) don't block draining the system jobs, so they may run concurrently with the system jobs being drained. With the `-ignore-system` flag, the system jobs are never drained, which makes them available during shutdown.

The `-ignore-system` flag appears to be strictly more useful than not having it set. Some example scenarios where you'd want to have a system job that runs until all other workloads have completed:
* log shippers
* monitoring agents
* ingress proxies (ex. Nginx or HAProxy in front of web services)
* CSI node plugins

Changing the default behavior would break backwards compatibility, so we want to solicit feedback from the community about whether this change would be disruptive.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `nomad node drain` command and its `-ignore-system` documentation, then review the linked pull request discussion and the documented current behavior. Done means reaching and recording a community-supported decision about changing the default, including the backward-compatibility impact.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.