Mark processes with asymmetric network errors as degraded

We can detect a process has an asymmetric network problem if it cannot connect to a process the failure monitor is reporting as healthy.

If the failure monitor reports a process as healthy, it means that process is communicating with the cluster controller properly. If a process cannot communicate with is being reported as healthy, it means something is happening in the network that the cluster controller is not detecting.

In certain scenarios this logic can cause spurious reports. For instance, if a transaction log is having network problems, but can still communicate with the cluster controller, we will mark all of our proxies as having problems. Because of this, machines which report asymmetric network errors should only be less preferred for being recruited, instead of trying to completely avoid using them.

With enough spare capacity this will generally allow the cluster to avoid recruiting the offending machine or process. In the case of the bad transaction log, both the transaction log and all the proxies would report problems. We would not have enough capacity to avoid using all the proxies, but we would have the ability to avoid the transaction log, because it is the only one of this process class reporting the problem.

This approach does not complete eliminate the risk that asymmetric network errors can cause a cluster outage, but will help greatly reduce their impact without a complete overhaul of our failure monitoring system.