hashicorp / hashicorp/consul

Document how requests from clients->servers are handled when some servers are in failed state

Open
#10,739 1 comment 0 reactions 0 assignees View on GitHub
theme/internals type/docs
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

When one or more servers in a cluster end up in a `failed` state (whether from a crash, network unreachability or "ungracefully" leaving/being shut down), our docs don't currently describe the behavior of how requests may or may not be routed to a `failed` server.

One point to keep in mind is that `failed` is typically assumed to be a temporary state, where a crashed server is expected to restart successfully or a network partition is expected to eventually heal, and servers may stay in `consul members list` in a `failed` state for up to 72 hours unless removed with [`consul force-leave`](https://www.consul.io/commands/force-leave).

We've seen reports of failed requests showing up in logs, but it's unclear if that's because Serf gossip on a client is still seeing a server as healthy, or if requests are intentionally still round-robined to `failed` servers with the expectation they will eventually return, and if these requests ultimately return errors or timeouts to a Consul client, or if these are transparently retried and can be expected to always return a response to the caller.

Contributor guide

Open the contributing guide

Research direction

Start with the request-routing behavior described in this issue, then compare it with the `consul members list` and `consul force-leave` behavior and the role of Serf gossip. Document what happens when servers are `failed`, including whether requests reach them, whether failures return errors or timeouts, and whether requests are retried transparently.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, documentation, networking
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.