Multiple failed processes rejoining the cluster can cause availability loss

When a storage process gets disconnected from the cluster, the cluster builds up a backlog of work for it. When the process rejoins after a long separation (but before it has been fully re-replicated), it quickly fills up its queue with that backlog. When this happens, ratekeeper is happy to ignore the process if it there aren't any other misbehaving processes in other fault domains.

If this happens to processes in separate fault domains simultaneously, however, ratekeeper will clamp down on the cluster until one of them manages to catch back up.

Part of this behavior comes from the fact that the cluster treats failed processes differently from lagging processes. The distinction fails a bit in this case, as these "lagging" processes aren't really lagging in the sense that was intended, and it probably makes more sense to still think of them as failed as they were before rejoining. @etschannen suggests there may be a way for us to redefine lagging processes as a sub-type of failure, in which case the logic for ratekeeper could be updated to ignore failures according to the replication policy.