Document how requests from clients->servers are handled when some servers are in failed state
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
When one or more servers in a cluster end up in a `failed` state (whether from a crash, network unreachability or "ungracefully" leaving/being shut down), our docs don't currently describe the behavior of how requests may or may not be routed to a `failed` server.
One point to keep in mind is that `failed` is typically assumed to be a temporary state, where a crashed server is expected to restart successfully or a network partition is expected to eventually heal, and servers may stay in `consul members list` in a `failed` state for up to 72 hours unless removed with [`consul force-leave`](https://www.consul.io/commands/force-leave).
We've seen reports of failed requests showing up in logs, but it's unclear if that's because Serf gossip on a client is still seeing a server as healthy, or if requests are intentionally still round-robined to `failed` servers with the expectation they will eventually return, and if these requests ultimately return errors or timeouts to a Consul client, or if these are transparently retried and can be expected to always return a response to the caller.
Contributor guide
Research direction
Start with the request-routing behavior described in this issue, then compare it with the `consul members list` and `consul force-leave` behavior and the role of Serf gossip. Document what happens when servers are `failed`, including whether requests reach them, whether failures return errors or timeouts, and whether requests are retried transparently.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, documentation, networking
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100