`RPC request for DC is currently failing` error for deleted wan federation cluster.

#### Overview of the Issue

I have a 'infrastructure' consul cluster ('central'), which has many 'local' clusters connected with wan federation (`consul join -wan ...`).
I started investigation of growing `raft.db` file on one of the central consul server node up to 10 Gb.
Raft db file size on other nodes in central cluster was about 50 mb. Also `consul snapshot` generated only 2.5 Mb of data.

I started to check servers logs, and I found vast amount of `RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes` errors in central consul servers logs. Tens of log lines generated each second on each of the servers.

I tried to setup one-node cluster with `datacenter=iron-prod2`, wan-joined it to central cluster, and logs on central cluster disappeared.
Then I tried to do `consul leave` on this `iron-prod2` node, and after its graceful shutdown, logs on central cluster appeared again.

### Consul for Servers

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 27de64da
version = 1.10.0
consul:
acl = disabled
bootstrap = false
known_datacenters = 9
leader = false
leader_addr = 192.168.132.115:8300
server = true
raft:
applied_index = 66293930
commit_index = 66293930
fsm_pending = 0
last_contact = 17.422082ms
last_log_index = 66293930
last_log_term = 126255
last_snapshot_index = 66293385
last_snapshot_term = 126255
latest_configuration = [{Suffrage:Voter ID:b088c336-6f39-ee06-6263-fdb12f112810 Address:192.168.132.115:8300} {Suffrage:Voter ID:b525db07-b17a-2d4c-47b9-ad7b339730f9 Address:192.168.132.69:8300} {Suffrage:Voter ID:aa120c96-6569-5f91-85ad-c3ea29edda8f Address:192.168.132.68:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 126255
runtime:
arch = amd64
cpu_count = 2
goroutines = 198
max_procs = 2
os = linux
version = go1.16.5
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 6216
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 16475
members = 7
query_queue = 0
query_time = 32
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 10011
members = 28
query_queue = 0
query_time = 2
```

### Operating system and Environment details
```
CentOS Linux release 7.3.1611 (Core)
Derived from Red Hat Enterprise Linux 7.3 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.3.1611 (Core)
CentOS Linux release 7.3.1611 (Core)
cpe:/o:centos:centos:7
```

### Log Fragments
Captured with TRACE log level.

Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.277Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.293Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.308Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.323Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.339Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.354Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.368Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.384Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.399Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.414Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.433Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.448Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.464Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.479Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.497Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.511Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.526Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.542Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.562Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.576Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.591Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.608Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.624Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.644Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.661Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.682Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.696Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.710Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.726Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.743Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.757Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.771Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.785Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.799Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.815Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.832Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:07 consul-infra1-01 consul[47027]: 2021-07-29T09:46:07.847Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.282Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.300Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.316Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.331Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.348Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.364Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.393Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.411Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.426Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.454Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes
Jul 29 09:46:08 consul-infra1-01 consul[47027]: 2021-07-29T09:46:08.472Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=iron-prod2 method=Health.ServiceNodes

Reproduce the central/local WAN-federation setup using `consul join -wan`, then remove the local cluster with `consul leave`. Start at the RPC path handling `Health.ServiceNodes` and inspect the repeated log behavior and raft.db growth. Done means deleting the WAN-federated cluster no longer causes continuous failed RPC requests or unbounded raft state.

Written by the indexing model from the issue text.