3 node LAN server starts reporting "ACL not found" for tokens created by WAN servers after working fine for months

#### Overview of the Issue

We get intermittent errors with clients such as consul-template after it working just fine for a long time. We have a 3 node WAN consul cluster that we have our 3 node LAN clusters replicate ACL. The ACL tokens are always generated by the WAN cluster and used to authenticate to both WAN and LAN clusters. This setup works most of the time but every once in a while the LAN servers will stop recognizing ACL tokens created by the WAN server. Many tokens created previously continue to function as expected, just new tokens fail. The issue persists until we restart all LAN consul servers. Everything goes back to normal but it will inevitably happen again in the future. We see this issue once every few months on one of our many LAN servers but production has been stable for well over a year until today. It is very infrequent but just today caused a production outage. We don't have steps to reproduce because it seems to happen at random on servers that have been running perfectly fine for a long time when we haven't changed any settings. I checked the logs and didn't see any logs related to syncing ACLs or other unusual activity.

#### Reproduction Steps

Steps to reproduce this issue, eg:

Unknown... this is an intermittent issue that seems to happen at random intervals every few months.

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 6
services = 11
build:
prerelease =
revision = c976ffd2
version = 1.10.3
consul:
acl = enabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 6
goroutines = 122
max_procs = 6
os = linux
version = go1.16.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 335
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 16253
members = 51
query_queue = 0
query_time = 1
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = c976ffd2
version = 1.10.3
consul:
acl = enabled
bootstrap = false
known_datacenters = 4
leader = false
leader_addr = :8300
server = true
raft:
applied_index = 19914996
commit_index = 19914996
fsm_pending = 0
last_contact = 9.430442ms
last_log_index = 19914996
last_log_term = 2208
last_snapshot_index = 19911799
last_snapshot_term = 2206
latest_configuration = [{Suffrage:Voter ID:a4578bc6-a74b-d727-e12e-6598154abee7 Address::8300} {Suffrage:Voter ID:58a9bf18-3214-b904-6d15-b4e549be8b0b Address::8300} {Suffrage:Voter ID:4f59b223-091b-1da3-d352-ae4a664455cd Address::8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 2208
runtime:
arch = amd64
cpu_count = 6
goroutines = 348
max_procs = 6
os = linux
version = go1.16.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 335
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 16253
members = 51
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 4930
members = 8
query_queue = 0
query_time = 1
```

### Operating system and Environment details

Runs in docker on Centos 7.

### Log Fragments

We don't have TRACE logs since we would have to leave it on for months at a time to see the error. The error we get from consul-template connecting to the LAN server is `health.service(peer.kube-etcd|passing): Unexpected response code: 403 (rpc error making call: ACL not found) (retry attempt 3 after "1s")`

There are no relevant logs on the server. On the client we see:
```
2022-06-29T15:51:48.692Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=:8300 error="rpc error making call: ACL not found"
```