Determining which processes are relatively safe to take down without excluding them

We have maintenance operations where we want to shut down FDB processes without moving data. For instance, if we're doing a rolling reboot of all the hosts in the fleet to install new patches at the OS level, we don't want to have to evacuate each host, because it would make the maintenance take too long. As another example, in the Kubernetes operator we have to delete and recreate pods in order to update environment variables or other parts of the pod spec. We do this instance-by-instance, but we want to make sure we only lose one unit of fault tolerance at a time. We can use the fault tolerance of the cluster to see if the cluster can lose a fault domain, but there are cases where this is too conservative. If we have one disk that has failed, this kind of check will prevent us from taking any hosts down for maintenance. It should be safe to take down any hosts that does not share replicas with the failed disk, because it would not reduce fault tolerance any further.

I think we can improve this by adding a check to the database to determine which processes are safe to take down. The full details of this feature will require more discussion, and probably a dedicated design doc, but I think we should be able to specify the desired fault tolerance after the processes get taken down, and should return all the processes that can be taken down without going below that level of fault tolerance. For instance, if we normally ran a cluster with a fault tolerance of 2, and we had one disk down, we could say `removable_processes fault_tolerance=1`, and it would return all processes that can be immediately shut down without dropping fault tolerance below 1. If the cluster was fully replicated, this list would return every process in the cluster. If some data were not fully replicated, it would return only the processes which could be taken down without dropping the fault tolerance to 0.

We'll also need to think about how this should work for multi-DC clusters. For instance, if we are running with a fault tolerance of 1 in two different DCs, then we may be happy to shut down 1 replica in each of the two DCs, but wouldn't be as happy to shut down both replicas in only one of the two DCs. On the other hand, we could be fine to be more conservative and say that we're only willing to lose 1 replica in either of the two DCs before we block planned maintenance.

No files, tests, or entry points are named. Start by reviewing the existing fault-tolerance and process-removal behavior, then write a design covering the requested fault-tolerance threshold and multi-DC semantics. Done means the behavior is agreed on and the design identifies the database interface and validation needed.

Written by the indexing model from the issue text.