cockroachdb / cockroachdb/cockroach
kvserver: reconsider gRPC server-side keepalive timeout
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The gRPC server-side keepalive timeout defaults to 2 seconds, via `COCKROACH_NETWORK_TIMEOUT`. This sets `TCP_USER_TIMEOUT` on the socket, instructing the OS to close connections if a send remains unacknowledged by the peer for 2 seconds.
We would ideally want this on the gRPC client side instead, since that would more quickly detect network outages and allow the client to fail over to a different node, reducing the duration of unavailability. Unfortunately, gRPC does not allow client-side keepalive timeouts below 10 seconds. Instead, we use application-level client-side RPC heartbeats, but unlike `TCP_USER_TIMEOUT` these are subject to head-of-line blocking in the RPC stack (including effects like scheduling latency). We therefore have to use a large RPC heartbeat timeout of 6 seconds to tolerate heavy loads in high-latency clusters (see #93397).
Given this, it isn't clear that an aggressive server-side gRPC timeout provides much benefit. It doesn't help in the case of a network outage -- the server is unlikely to care about the connection remaining open, and the client has to wait for its own timeout to fire. It would more quickly detect asymmetric partitions, allowing the dialback mechanism to promote it to a symmetric partition (see #94778), but this is likely rare enough that a higher timeout is ok. Under persistently high latency it might be helpful to eagerly close the connection and let the client fail over to a lower-latency node, but this assumes the lease also moves and that the client won't block and re-dial the node (mostly true in 23.2 with #99191). It can also be detrimental since it can cause additional disruption with transient latency spikes like intermittent packet loss.
We should reconsider the value for the server-side keepalive timeout, subject to availability benchmarks (especially with asymmetric partitions and latency spikes).
See also #93007.
https://github.com/cockroachdb/cockroach/blob/e97fe41e72fcb61dfda3ad5d6e73010311ce002e/pkg/rpc/keepalive.go#L26-L47
Jira issue: CRDB-30871
Contributor guide
Assessment
This issue has not been assessed yet.