hashicorp / hashicorp/consul

Server should report when they are not fully initialized

Open
#10,968 0 comments 2 reactions 0 assignees View on GitHub
theme/reliability
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

Related to #10967

When a server process starts it will accept RPC connections before it is ready to handle them. The server must connect to other servers, and may need to receive a snapshot or logs from them. Then it has to populate the FSM from that snapshot. This can take some time when the snapshot is large.

Either servers should not accept connections until they are initialized (e.g., caught up to the latest Raft index), or they should report a specific error, so that clients know they need to communicate with a different server.

Normally this would be accomplishing by not listening on the port until we are ready to handle requests. That doesn't work for consul because both Raft RPC and Consul RPC use the same port. I assume that the raft peer needs to accept raft RPC requests to complete initialization of the peer (would be worth confirming though), so we likely do have to listen on the port before the server is ready to handle RPC requests.

One way to solve this would be to refuse any Consul RPC requests (`Server.handleConsulConn`) until the raft peer has finished initializing. I'm not sure if any of [the other services behind this RPC port](https://github.com/hashicorp/consul/tree/main/docs/rpc) require similar handling or not.

Contributor guide

Open the contributing guide

Research direction

Start with Server.handleConsulConn and the RPC services listed in docs/rpc. Confirm whether Raft RPC must remain available while a server catches up, then define how Consul RPC requests should report that initialization is incomplete. Done means clients can distinguish an unready server and the behavior is consistent with the chosen connection-handling approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.