Separate distributed silo health checks from cluster membership

This is a proposal based on the discussion in #2542 and #2569.

The original assumption about 'outsourcing' cluster membership to external providers like SF and Docker orchestration managers was that a membership oracle plug-in for a system like that would completely replace our existing cluster membership logic. As we started working on a couple of plug-ins, for SF and Docker, it became clear that there is a functionality gap that we didn't think through originally.

Each of these cluster management systems can detect node failures. SF can also detect network partitions between nodes. It's unclear yet to what extend Swarm/Kubernetes/Mesos can do that. However, these detection mechanisms are not sufficient for detecting certain kinds of node failures that are fairly rare in practice but not impossible. For example, a bug in our messaging code may render a silo unavailable to other silos in the cluster while SF/Swarm/Kubernetes/Mesos would consider the silo node healthy. Our current cluster membership protocol handles such cases well, and it would be bad to lose this level of failure detection guarantees.

We detect unavailability of silos today by having each silo periodically ping a configurable number of other silos. We deterministically decide who's pining whom, and handle special cases of small clusters. When a failure of a node is suspected, that suspicion is fed into the cluster membership protocol as a death vote. Votes are persisted (with configurable expiration) in the abstraction of membership table.

The proposal is to decouple the failure detection logic from cluster membership into a separate layer/service, so that it would continue to work as is in the default cluster membership case, but would also integrate with the external cluster management solutions. That new layer would play the role of a distributed silo health check service (HCS). It would consume from the cluster management (CM) layer a list of silos in the cluster, and, ideally, notifications about any changes to cluster state. Worst case, the HCS would have to poll CM to refresh. Instead of feeding back silo death votes, like it does today, HCS would send silo death verdicts, so that CM would stop/restart respective nodes. This requires a solution for storing votes to aggregate them before a verdict is reached.

Note that HCS would act almost as an external canary service detecting application level failures. That would not interfere with whatever node failure detection methods CMs employ because in the worst case they will reach the same node death verdict in the end. At the same time, by being integrated with CM, HCS can provide better detection of horizontal communication failures within a cluster than an external canary service with no knowledge of the cluster state.

Tactically, I suggest we do the following:

Step 1. first merge #2542 as is. The justification here is that by monitoring for network partitions SF already covers 98% of node availability issues. People that want to run Orleans on SF will have a ready to use solution that meets their expectations for other SF-based services.

Step 2. Refactor/relayer the CM-HCS relation with the default end behavior unchanged.

Step 3. Integrate HCS with the SF CM.

Step 4. Can go in parallel with 3. Integrate HCS with Docker CM. The concern about Docker is that it won't even detect basic network partitions, and those are more frequent in production than esoteric messaging bugs. Hence, it seems dangerous to have a cluster membership option for Docker without the HCS that would detect those cases as well.

Start by reading discussions #2542 and #2569, then trace the existing cluster membership protocol's silo pings, death votes, and membership table. The issue names no files or tests; clarify the HCS/CM boundary and review the proposed four-step sequence before implementation can begin.

Written by the indexing model from the issue text.