Floating point exception when excluding stateless processes 7.1.28

We experienced a `Floating point exception` in FoundationDB while excluding stateless processes from the cluster.

Cluster details:
- FoundationDB 7.1.28
- SSD storage engine, double redundancy
- 36 machines
- 713 processes in total

10 stateless processes per machine were added in error. We began to remove them per machine, excluding each process one at a time per each machine. At node 27, the floating point exception appeared in `fdbcli`:

```
fdb> exclude 10.31.2.88:4579

WARNING: Long delay (Ctrl-C to interrupt)

The database is unavailable; type `status' for more information.

SIGNAL: Floating point exception (8)
Trace: addr2line -e fdbcli.debug -p -C -f -i 0x7ff6d4669980 0xaf2136 0xbcc750 0xbccc60 0xbcd22a 0xbcd486 0xbc9408 0xbc9943 0xbc9d3c 0xbcfda8 0x849a10 0xce1910 0xce1cfb 0x6dda80 0xd91086 0x8adc92 0xc4a5cf 0x869071 0x537923 0x7ff6d4287c87
Floating point exception
```

The database flipped to unavailable and didn't come back until we re-included the stateless processes and added a process with `role = data_distributor`.

This is a snippet of one of the tracefiles:
```

```

I pulled down the debug binaries and ran `addr2line` against them.

### fdbcli

```
$ addr2line -e fdbcli.debug.x86_64 -p -C -f -i 0x7ff6d4669980 0xaf2136 0xbcc750 0xbccc60 0xbcd22a 0xbcd486 0xbc9408 0xbc9943 0xbc9d3c 0xbcfda8 0x849a10 0xce1910 0xce1cfb 0x6dda80 0xd91086 0x8adc92 0xc4a5cf 0x869071 0x537923 0x7ff6d4287c87
?? ??:0
JSONDoc::at(std::string, bool) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/JSONDoc.h:285
(inlined by) JSONDoc::operator[](std::string) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/JSONDoc.h:288
(inlined by) a_body1cont8 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/SpecialKeySpace.actor.cpp:1002
getClientDatabaseStatus(JSONDoc, JSONDoc) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/StatusClient.actor.cpp:502
(anonymous namespace)::StatusFetcherImplActorState<(anonymous namespace)::StatusFetcherImplActor>::a_body1cont7(int) [clone .isra.1454] at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/StatusClient.actor.cpp:606
ActorCallback<(anonymous namespace)::StatusFetcherImplActor, 1, Optional >::error(Error) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1323
Callback::remove() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:400
(inlined by) a_exitChoose3 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/StatusClient.actor.g.cpp:2232
(inlined by) a_callback_fire at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/StatusClient.actor.g.cpp:2283
(inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1321
(anonymous namespace)::TimeoutMonitorLeaderActorState<(anonymous namespace)::TimeoutMonitorLeaderActor>::a_body1loopBody1(int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/StatusClient.actor.g.cpp:2484 (discriminator 14)
ISimulator::ProcessInfo::global(int) const at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/simulator.h:177
(inlined by) ISimulator::global(int) const at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/simulator.h:461
(inlined by) FlowTransport::transport() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/FlowTransport.h:267
(inlined by) a_body1cont2 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/genericactors.actor.h:397
(inlined by) a_body1when1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/genericactors.actor.g.h:6268
(inlined by) a_callback_fire at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/genericactors.actor.g.h:6289
(inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1321
SAV::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:696
(inlined by) a_body1cont3 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/StatusClient.actor.g.cpp:992
(inlined by) a_body1cont7 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/StatusClient.actor.g.cpp:1014
(anonymous namespace)::WaitValueOrSignalActorState >::a_body1loopBody1Catch1(Error const&, int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/genericactors.actor.h:386 (discriminator 4)
SAV::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:696
(inlined by) void SAV::sendAndDelPromiseRef(StatusReply&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:690
(inlined by) NetSAV::receive(ArenaObjectReader&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/fdbrpc.h:110
Peer::onIncomingConnection(Reference, Reference, Future) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/FlowTransport.actor.cpp:902 (discriminator 1)
Reference::operator=(Reference const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/FastRef.h:131
(inlined by) TransportData::getOrOpenPeer(NetworkAddress const&, bool) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/FlowTransport.actor.cpp:1425
void SAV::send(Void&&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:662
N2::Net2::run() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Net2.actor.cpp:1450
runNetwork() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/NativeAPI.actor.cpp:2612
internal_thread_helper::DoOnMainThreadVoidActor1State >::a_body1cont1(Void const&, int) [clone .isra.334] at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/ThreadHelper.actor.g.h:305
(inlined by) ~DoOnMainThreadVoidActor1 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/ThreadHelper.actor.h:50
(inlined by) a_body1Catch1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/ThreadHelper.actor.g.h:332
(inlined by) a_body1cont2 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/ThreadHelper.actor.g.h:452
(inlined by) a_body1cont5 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/ThreadHelper.actor.g.h:480
(inlined by) a_body1cont1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/ThreadHelper.actor.g.h:352
__gnu_cxx::__normal_iterator > >::__normal_iterator(unsigned long* const&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_iterator.h:784
(inlined by) std::vector >::begin() at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_vector.h:699
(inlined by) MultiVersionApi::runNetwork() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:2292
main at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbcli/fdbcli.actor.cpp:2295
?? ??:0
```

### fdbserver

```
$ addr2line -e fdbserver.debug.x86_64 -p -C -f -i 0x363477c 0x2c5e80c 0x2c61ca2 0x1f0b987 0x1069e59 0x1029541 0x1032fd0 0x103331c 0x1f30d64 0x1f30fee 0xc14ae7 0x3442cd0 0x34430bb 0x1130460 0x35d0cb6 0x9d32b9 0x7f9f4492dc87
int const& std::min(int const&, int const&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_algobase.h:200
(inlined by) operator<(StringRef const&, StringRef const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Arena.h:807
(inlined by) std::less >::operator()(Standalone const&, Standalone const&) const at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_function.h:386
(inlined by) std::_Rb_tree, std::pair const, std::unique_ptr > >, std::_Select1st const, std::unique_ptr > > >, std::less >, std::allocator const, std::unique_ptr > > > >::_M_lower_bound(std::_Rb_tree_node const, std::unique_ptr > > >*, std::_Rb_tree_node_base*, Standalone const&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h:1888
(inlined by) std::_Rb_tree, std::pair const, std::unique_ptr > >, std::_Select1st const, std::unique_ptr > > >, std::less >, std::allocator const, std::unique_ptr > > > >::lower_bound(Standalone const&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h:1203
(inlined by) std::map, std::unique_ptr >, std::less >, std::allocator const, std::unique_ptr > > > >::lower_bound(Standalone const&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_map.h:1239
(inlined by) std::map, std::unique_ptr >, std::less >, std::allocator const, std::unique_ptr > > > >::operator[](Standalone const&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_map.h:495
(inlined by) DynamicEventMetric::registerFields(MetricKeyRef const&, std::vector, std::allocator > >&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/TDMetric.cpp:218
IFailureMonitor::failureMonitor() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/FailureMonitor.h:132
(inlined by) a_body1loopBody1 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/LoadBalance.actor.h:623
Callback::remove() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:397
(inlined by) a_exitChoose2 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/NativeAPI.actor.g.cpp:17941
(inlined by) a_callback_error at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/NativeAPI.actor.g.cpp:17977
(inlined by) cancel at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/NativeAPI.actor.g.cpp:18275
Promise::~Promise() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:930
(inlined by) NotifiedQueue::~NotifiedQueue() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:983
(inlined by) NetNotifiedQueue::~NetNotifiedQueue() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/fdbrpc.h:646
(inlined by) NetNotifiedQueue::~NetNotifiedQueue() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/fdbrpc.h:646
(inlined by) NetNotifiedQueue::destroy() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/fdbrpc.h:654
(inlined by) NotifiedQueue::delPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1046
FieldHeader::update(long const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/TDMetric.actor.h:381
(inlined by) FieldLevel, FieldValueBlockEncoding >::log(long, unsigned long, bool&, long&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/TDMetric.actor.h:496
std::vector >::_M_check_len(unsigned long, char const*) const at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_vector.h:1646
(inlined by) void std::vector >::_M_range_insert<__gnu_cxx::__normal_iterator > > >(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, std::forward_iterator_tag) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/vector.tcc:718
(inlined by) void std::vector >::_M_insert_dispatch<__gnu_cxx::__normal_iterator > > >(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, std::__false_type) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_vector.h:1549
(inlined by) __gnu_cxx::__normal_iterator > > std::vector >::insert<__gnu_cxx::__normal_iterator > >, void>(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_vector.h:1270
(inlined by) void LogPushData::addTags > >(std::vector >) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/LogSystem.h:753
(inlined by) a_body1loopBody1cont1 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/CommitProxyServer.actor.cpp:423
(inlined by) a_body1loopBody1break1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/CommitProxyServer.actor.g.cpp:1511
(inlined by) a_body1loopBody1loopBody1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/CommitProxyServer.actor.g.cpp:1470
SAV::get() const at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:652
(inlined by) Future::get() const at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:796
(inlined by) a_body1cont1 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/genericactors.actor.h:1057
CommitBatch::PostResolutionActorState::a_body1cont11cont1(int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/CommitProxyServer.actor.cpp:1332
(anonymous namespace)::CreateAndLockProcessIdFileActorState<(anonymous namespace)::CreateAndLockProcessIdFileActor>::a_body1loopBody1cont2(int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/worker.actor.cpp:2677
SAV > >::delFutureRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:743
(inlined by) Future > >::~Future() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:832
(inlined by) StrictFuture > >::~StrictFuture() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:894
(inlined by) a_body1loopBody1 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:894
```

I see there are OOM's in the tracefiles which I also see in our charts. I'm wondering if this is a known or expected issue?

Start by reproducing the failure through the fdbcli `exclude` command using a cluster with stateless processes being removed. Review the supplied fdbcli crash trace and server tracefile, including the state where no data_distributor process is available. Done means excluding processes no longer causes a floating point exception or leaves the database unavailable.

Written by the indexing model from the issue text.