Detect and handle connections with excessive latency

We've recently added some targeted features (e.g. #3891, #3686) that seek to detect when certain types of connections experience high latencies and then reset those connections. This is intended to address a specific failure scenario that is resolved by reestablishing these connections, and it is done in paths where the high latencies are particularly impactful.

It is possible, though, that we could experience high latencies over any connection, and the cause of those latencies may not be addressed by resetting the connection. We should try to generalize this work to make this detection function more broadly and to apply additional measures to correct it (e.g. excluding processes).

One of the challenges for detection is to make sure we can tolerate different topologies that result in different expected latencies between processes. For example, specifying a fixed latency threshold would need to accommodate processes that are intentionally set up far apart, such as with a multi-region cluster, in which case it may no longer be sensitive enough.

Likewise, taking corrective action needs to be done more carefully as we attempt more significant interventions. Resetting a connection is low impact and can be done by a process unilaterally, but excluding processes should be done intelligently to avoid negative effects.