Clarify documentation around `consul leave` `leave_on_terminate` and upgrades
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
The current guidance around when `consul leave` should be used is unclear. There are at least three different parts of our documentation that provide hints about the role and purpose of the "leave" process in general:
- [`consul leave` CLI docs](https://www.consul.io/commands/leave)
- [`leave_on_terminate` option docs](https://www.consul.io/docs/agent/options#leave_on_terminate)
- [upgrade overview](https://www.consul.io/docs/upgrading#standard-upgrades) which recommends using `consul leave` during a rolling restart (i.e. when servers are only restarting temporarily.
In general, we are ambiguous about the term "graceful shutdown" (which typically refers to servers completing inflight requests) and other properties of our agents such as explicitly giving up their membership of the gossip cluster or removing themselves as a voting member of the raft quorum.
The actual behaviour of `consul leave` (or `leave_on_terminate`) is therefore a little hard to interpret as to whether it is the optimal or even correct thing to use in specific situations:
- decommissioning an agent (client or server) forever (e.g. as part of an elastically scaled cluster or to be replaced by a different server with different host/ip/state in immutable infra pattern)
- restarting the agent with minimal disruption (e.g. in an in-place binary upgrade on either a client or server agent)
- temporarily shutting down an agent for maintenance with the intention to bring it back up later.
It's _arguable_ that the current behaviour of leave is appropriate even if not necessary for all of these cases, however it's certainly unclear from our current documentation whether it's _supposed_ to be used in each of these cases or even what the tradeoffs are of using it vs. not.
It seems like during a rolling restart of servers, it would be ideal _not_ to reconfigure raft to remove each server from the quorum one-by-one (which is what leave will do per our current recommendation). The operator is not in this case explicitly requesting a reduction in the quorum of servers so it could be argued that it's wrong to recommend a leave. However the downsides of this may not be significant in practice provided only one server is offline at once as the reduction in quorum from one node leaving shouldn't impact fault tolerance properties.
On the other hand, having even temporary restarts fully remove the agent from gossip and raft and re-introduce them, might reduce spurious errors where the loss of availability is detected by gossip or autopilot and alerted on. Recommending against `leave` or changing it's behaviour could actually be worse in terms of operator UX or even causing cascading impact due to things being marked as unhealthy during the restart.
So we may decide to change the behaviour of `leave` or leave it for the full-decommission use case and augment it with other commands or alternate behaviour to cover the other use-cases above. Or we may choose to leave things as are they are and just clarify in our documentation how to correctly use `leave` and any possible tradeoffs to consider.
Contributor guide
Research direction
Compare the consul leave CLI docs, leave_on_terminate option docs, and the standard-upgrades overview cited in the issue. Clarify the intended guidance for decommissioning, rolling restarts, and temporary maintenance, including the tradeoffs involving gossip membership and Raft quorum; document any behavior decision if the existing guidance is insufficient.
Written by the indexing model from the issue text.
Assessment
- Domain
- devops, distributed-systems, documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100