hashicorp / hashicorp/nomad

Nomad should allow draining all clients which matches metadata values

Open
#1,037 2 comments 3 reactions 0 assignees View on GitHub
theme/drain type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Operators could drain clusters easily if Nomad allowed specifying certain values which matches metadata while draining. For example, operators could drain all nodes in an AWS ASG, or all nodes which matches the value of Nomad's client's version.

The implementation can be broken down into 2 phases which can be implemented independently:

1. Batch Drain API - An alternative API to [`/v1/node/:nodeid/drain`](https://www.nomadproject.io/api-docs/nodes#drain-node) that accepts multiple Node IDs *and updates those nodes atomically in Raft.*
2. Selection CLI/UI ... possibly API? - The Batch Drain API may or may not include a selection API. This may be implemented purely in the CLI/UI or even extrernal tooling. Eventually a general selection solution would be ideal as it would be broadly useful across Nomad (eg a selection DSL for list APIs). For very large clusters there's a slight optimization to sending a query vs independent IDs. However even at 10k nodes, draining every node would take ~400 KB. A large request, but not a concern.

While many users have made their own solution to `#2`, without `#1` being solved it risks races between marking nodes as draining and rescheduling work causing lots of churn.

Imagine the hypothetical command:

```
nomad node-drain -enable -yes ab12 cd34 ef56
```

**Without** a batching API that could produce the following timeline:

1. /v1/node/ab12/drain
2. ab12 set to drain
3. allocs on ab12 rescheduled onto cd34 ef56
4. /v1/node/cd45/drain
5. allocs from ab12 must be rescheduled again
6. etc

As you can see we may end up rescheduling allocations multiple times. While the cluster always stabilizes there will be an unnecessary amount of work with lots of allocations created and replaced in a short amount of time.

**With* a batching API all of the nodes would be atomically updated and allocations would be rescheduled exactly once.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the existing /v1/node/:nodeid/drain API and the proposed nomad node-drain command, then review how node updates are applied through Raft. Define the batch operation's atomic behavior and how node selection should be handled; done means multiple nodes can be drained without the rescheduling races described here.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, cli, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.