Don’t kill allocations that have not yet been replaced

### Nomad version
Nomad v0.8.7 (21a2d93eecf018ad2209a5eab6aae6c359267933+CHANGES)

### Operating system and Environment details
GNU/Linux

### Issue
When heart-beat messages sent by Nomad clients to the server cluster are missed, the cluster will consider allocations running on these clients lost and will attempt to reschedule them. This may trigger, e.g., due to connectivity issues between a datacenter hosting Nomad clients (DC2), and another datacenter hosting the server cluster (DC1). Now, if lost allocations were defined as part of a job that specifies that tasks must be run in DC2, allocations will be queued until connectivity to DC2 returns. At this point, Nomad will instantly kill all old allocations despite the fact these allocations have not yet been replaced, causing a service outage.

I wonder whether this case could be handled more gracefully on the part of Nomad. I realize that there may be issues related to which agent comes back in contact when. E.g., a vacant node in DC2 might re-establish contact to DC1 prior to the node running the lost allocation, so that the replacement allocation will be dispatched before the lost allocation is re-discovered.

Would it make sense to establish some kind of grace period that applies when a datacenter has been wholly lost? This grace period would cause Nomad to give all clients time to re-establish contact to the cluster before any further steps are taken, which in turn would allow the cluster to establish that some or all of the lost allocations there are healthy and could be reinstated.

Alternatively, (in case reinstating lost allocations is not desirable for some reason), perhaps Nomad could still hold off from killing old allocations until replacement allocations are up and healthy.

### Reproduction steps

- Run a Nomad server cluster in DC1
- Run a set of Nomad clients in DC2
- Submit a job that is specified to run only in DC2
- Disconnect DC1 from DC2
- Reconnect DC1 to DC2

### Bonus question

Can you advise whether my understanding is correct in that client heart-beats must be sent to a quorum of servers, not just any server, for the client to be considered alive? In other words: would it make a difference if we place a Nomad server in DC2 that is in contact with clients during the time-period where DC2 is segmented off?

No files, tests, or entry points are identified. Start by reproducing the DC1/DC2 partition and reconnection sequence described in the issue, then trace how lost allocations are replaced and when old allocations are killed. Done means defining and validating a decision between a datacenter grace period and waiting for healthy replacements before termination.

Written by the indexing model from the issue text.