hashicorp / hashicorp/consul

"maintenance mode" concept for the node itself in addition to maint mode for services running on that node

Open
#11,761 2 comments 1 reaction 0 assignees View on GitHub
theme/consul-nomad type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Feature Description

It seems like "maintenance mode" is associated with a service (by associating the service with a dedicated "maintenance mode" failing health check) but not with the node itself. Means the node is still shown as alive in the cluster and nomad keeps scheduling allocations on that node. This tricky for a couple of things:
- even if the health check fails and nomad is used to take the service instance down, it may be rescheduled on the same node
- when it gets rescheduled with the same service name, it still is flagged in maintenance

The consul CLI also makes it really easy to enable maintenance mode for all services on a node and to associate maintenance mode with the state of a node, rather than a service.

Maintenance mode also disables service discovery for that instance (rightfully so) but nomad won't care about rescheduling it and this may lead, in clusters with few instances of the same service, to reduced capacity or complete unavailability. For example, in the case of a service that uses leader election and a health check to register the leader as the instance to route traffic to, this service could become completely undiscoverable (especially if leader election is done with consul kv record and locks that keep functioning in maintenance mode)

This seems like it could be solved simply by surfacing the "maintenance mode" at the node level:
- `consul members` could then be used to quickly track which nodes are in maintenance mode
- nomad could use a node health check and not schedule instances there (and migrate existing instances)

#### Use Case(s)

- nomad could avoid scheduling any service that is at risk of not being discoverable on that node
- nomad drain could trigger consul maintenance mode
- consul maintenance mode could trigger nomad drain of the node

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing Consul node and service maintenance behavior, including the `consul members` entry point, then examine how Nomad scheduling and `nomad drain` interact with node health. Define the node-level maintenance semantics, including existing allocations and service discovery, and document the expected Consul-to-Nomad and Nomad-to-Consul behavior when complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.