net: record linux-level TCP round-trip time and variance

We currently rely on our RPC layer's period ping-pong exchanges with other nodes to measure network latency. This does not work very well, since it also measures all kinds of things in our stack and in particular, CPU overload will always significantly inflate these measurements, even if the underlying connection latencies haven't really changed.

A set of metrics that much more tightly measures physical latency between the endpoints would be desirable. It would help in escalations, most recently in https://github.com/cockroachlabs/support/issues/3342.

The linux kernel maintains these already! We should use them.

See prototype https://github.com/cockroachdb/cockroach/pull/149956.

Here's a Cursor-generated synopsis of where this data comes from:

### How it Works: The `TCP_INFO` Syscall

The proposed method uses the standard Linux `getsockopt` syscall on an active TCP socket. By using the `TCP_INFO` option, we can ask the kernel to provide a detailed report on the state of a specific TCP connection.

This syscall returns a `struct tcp_info` populated with metrics about the connection's performance. For our purposes, we are interested in two key fields:

1. `tcpi_rtt`: The **Smoothed Round Trip Time (SRTT)**, measured in microseconds.
2. `tcpi_rttvar`: The **RTT variance**, also in microseconds, which measures the jitter or stability of the RTT.

A proof-of-concept has been implemented in `pkg/server/status/net_rtt_test.go` to demonstrate the feasibility of this approach.

### How the Linux Kernel Maintains RTT

The kernel's RTT values are not just raw, instantaneous measurements. They are carefully managed to provide a stable view of a connection's health.

- **RTT Sampling:** The kernel continuously takes new RTT samples by measuring the time between sending a TCP segment and receiving its corresponding acknowledgment (ACK).

- **Smoothed RTT (SRTT):** To avoid overreacting to transient network spikes, the kernel uses an **Exponentially Weighted Moving Average (EWMA)**. This algorithm, foundational to modern TCP, smooths out the RTT samples to produce a stable latency estimate. The `tcpi_rtt` value we read from the syscall is this smoothed average.

- **RTT Variance (RTTVAR):** The kernel also maintains a smoothed average of the *variance* of the RTT samples. This metric is crucial because it quantifies the stability of the connection's latency.

These two values are fundamental to TCP's performance, as they are used to dynamically calculate the **Retransmission Timeout (RTO)**—the time the kernel waits before assuming a packet is lost.

### Further Reading

For more detailed information, the following resources are recommended:

- **Linux Manual Page for TCP:** `man 7 tcp` provides the official documentation for the `TCP_INFO` socket option and all of its fields.
- **RFC 6298:** This document specifies the standard algorithm for computing TCP's Retransmission Timeout, detailing the logic for SRTT and RTTVAR.
- **Kernel Header File:** The exact C definition for `struct tcp_info` can be found in the Linux kernel source in ``.

Jira issue: CRDB-52438