cockroachdb / cockroachdb/cockroach

kvserver: reconsider gRPC server-side keepalive timeout

Open
#109,317 2 comments 0 reactions 0 assignees View on GitHub
C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The gRPC server-side keepalive timeout defaults to 2 seconds, via `COCKROACH_NETWORK_TIMEOUT`. This sets `TCP_USER_TIMEOUT` on the socket, instructing the OS to close connections if a send remains unacknowledged by the peer for 2 seconds.

We would ideally want this on the gRPC client side instead, since that would more quickly detect network outages and allow the client to fail over to a different node, reducing the duration of unavailability. Unfortunately, gRPC does not allow client-side keepalive timeouts below 10 seconds. Instead, we use application-level client-side RPC heartbeats, but unlike `TCP_USER_TIMEOUT` these are subject to head-of-line blocking in the RPC stack (including effects like scheduling latency). We therefore have to use a large RPC heartbeat timeout of 6 seconds to tolerate heavy loads in high-latency clusters (see #93397).

Given this, it isn't clear that an aggressive server-side gRPC timeout provides much benefit. It doesn't help in the case of a network outage -- the server is unlikely to care about the connection remaining open, and the client has to wait for its own timeout to fire. It would more quickly detect asymmetric partitions, allowing the dialback mechanism to promote it to a symmetric partition (see #94778), but this is likely rare enough that a higher timeout is ok. Under persistently high latency it might be helpful to eagerly close the connection and let the client fail over to a lower-latency node, but this assumes the lease also moves and that the client won't block and re-dial the node (mostly true in 23.2 with #99191). It can also be detrimental since it can cause additional disruption with transient latency spikes like intermittent packet loss.

We should reconsider the value for the server-side keepalive timeout, subject to availability benchmarks (especially with asymmetric partitions and latency spikes).

See also #93007.

https://github.com/cockroachdb/cockroach/blob/e97fe41e72fcb61dfda3ad5d6e73010311ce002e/pkg/rpc/keepalive.go#L26-L47

Jira issue: CRDB-30871

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.