ACL policies replication bug

#### Overview of the Issue
---

Hey team. We noticed that ACL Policy replication sometimes does not happen in clusters connected via WAN. We didn't see any errors or failed replication reports while encountering this bug. When it happens, the primary has an updated policy, and one or two secondaries have an old version of the policy (or do not create a policy if it's a new one). Policies don't sync during the next replication poll, and, ultimately, the consul cluster thinks it's in sync with the primary, while it's not true (indexes are correct).

Important: Replication works *most* of the time, it fails only from time to time and completely randomly (sometimes it happens during the first five tries, sometimes during 40). Replicated Index is up-to-date in all the cases. We see no errors in logs either during failed or successful replication. For testing, we used consul versions 1.16.6 and 1.15.10.

#### Reproduction Steps

Steps to reproduce this issue:

1. Create three clusters and connect them via WAN.
2. Create a testing policy on the primary cluster
3. Update the policy with slight changes (e.g. change service_prefix from test-1 to test-2....10) 1-15 times.
4. Read the policy on the secondary clusters.

### Consul info for Servers
Server info

```Output from client 'consul info' command here
consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = a8dca240
version = 1.15.10
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 3
leader = false
leader_addr = ip:8300
server = true
raft:
applied_index = 214
commit_index = 214
fsm_pending = 0
last_contact = 34.3041ms
last_log_index = 214
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:297216aa-1206-becd-a357-7fa6a5f64a2e Address:10.0.0.199:8300} {Suffrage:Voter ID:0dbca19c-0310-4eda-4595-6fa73fc91f93 Address:10.0.0.180:8300} {Suffrage:Voter ID:fbcef797-3d7a-d141-bd8c-cbbc36b3f358 Address:10.0.0.141:8300} {Suffrage:Voter ID:3c198304-4604-a298-7786-65f3cc2a241c Address:10.0.0.186:8300} {Suffrage:Voter ID:fbdfb4f5-f3fb-e089-4657-c06290938243 Address:10.0.0.217:8300}]
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 2
runtime:
arch = amd64
cpu_count = 8
goroutines = 176
max_procs = 8
os = linux
version = go1.21.7
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 5
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 221
members = 15
query_queue = 0
query_time = 1
```

We dont' have clients in this testing setup

```
Server agent HCL config
{
"client_addr": "{{ GetInterfaceIP \"eth1\" }} 127.0.0.1",
"bind_addr": "{{ GetInterfaceIP \"eth1\" }}",
"data_dir": "/consul/data",
"log_level": "TRACE",
"datacenter": "${DC}",
"encrypt": "${GOSSIP_KEY}",
"primary_datacenter": "${PRIMARY_CLUSTER}",
"retry_join_wan": ["${RETRY_DNS}:8302"],
"encrypt_verify_incoming": true,
"encrypt_verify_outgoing": true,
"leave_on_terminate": true,
"skip_leave_on_interrupt": false,
"tls": {
"defaults": {
"tls_cipher_suites": "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"verify_incoming": false,
"verify_outgoing": true,
"ca_file": "/consul/config/ca.cert.pem",
"cert_file": "/consul/config/consul.crt",
"key_file": "/consul/config/consul.key"
},
"internal_rpc": {
"verify_server_hostname": true,
"verify_incoming": true
}
},
"rpc": {
"enable_streaming": true
},
"ports": {
"http": -1,
"https": 8501,
"grpc": -1,
"grpc_tls": 8502
},
"auto_encrypt": {
"allow_tls": true
},
"acl": {
"enabled": true,
"default_policy": "deny",
"down_policy": "extend-cache",
"enable_token_persistence": true,
"enable_token_replication": true,
"tokens": {
"master": "${MASTER_TOKEN}",
"agent": "${MASTER_TOKEN}",
"replication": "${MASTER_TOKEN}"
}
},
"connect": {
"enabled": true
},
"telemetry": {
"statsd_address": "localhost:8125",
"disable_hostname": true
}
}

```

### Operating system and Environment details

We have clusters running on ECS Fargate, in the same VPC, SG rules allow all traffic (for debugging purposes).
Tried to disable encryption, verifications, and different consul versions (1.16.6 and 1.15.10). We tried multiple clusters in multiple AWS accounts with the same results. We also tried to use either a master token as a replication or a dedicated one only for replication.

WAN setup:
```
consul members -wan
Node Address Status Type Build Protocol DC Partition Segment
ip-10-0-0-134.region.compute.internal.consul-1-0 10.0.0.134:8302 alive server 1.15.10 2 consul-1-0 default
ip-10-0-0-145.region.compute.internal.consul-1-2 10.0.0.145:8302 alive server 1.15.10 2 consul-1-2 default
ip-10-0-0-146.region.compute.internal.consul-1-1 10.0.0.146:8302 alive server 1.15.10 2 consul-1-1 default
ip-10-0-0-151.region.compute.internal.consul-1-2 10.0.0.151:8302 alive server 1.15.10 2 consul-1-2 default
ip-10-0-0-154.region.compute.internal.consul-1-1 10.0.0.154:8302 alive server 1.15.10 2 consul-1-1 default
ip-10-0-0-155.region.compute.internal.consul-1-0 10.0.0.155:8302 alive server 1.15.10 2 consul-1-0 default
ip-10-0-0-165.region.compute.internal.consul-1-1 10.0.0.165:8302 alive server 1.15.10 2 consul-1-1 default
ip-10-0-0-167.region.compute.internal.consul-1-1 10.0.0.167:8302 alive server 1.15.10 2 consul-1-1 default
ip-10-0-0-176.region.compute.internal.consul-1-2 10.0.0.176:8302 alive server 1.15.10 2 consul-1-2 default
ip-10-0-0-178.region.compute.internal.consul-1-0 10.0.0.178:8302 alive server 1.15.10 2 consul-1-0 default
ip-10-0-0-186.region.compute.internal.consul-1-2 10.0.0.186:8302 alive server 1.15.10 2 consul-1-2 default
ip-10-0-0-196.region.compute.internal.consul-1-2 10.0.0.196:8302 alive server 1.15.10 2 consul-1-2 default
ip-10-0-0-210.region.compute.internal.consul-1-1 10.0.0.210:8302 alive server 1.15.10 2 consul-1-1 default
ip-10-0-0-211.region.compute.internal.consul-1-0 10.0.0.211:8302 alive server 1.15.10 2 consul-1-0 default
ip-10-0-0-216.region.compute.internal.consul-1-0 10.0.0.216:8302 alive server 1.15.10 2 consul-1-0 default
```
### Log Fragments

```
agent.server.replication.acl.policy: acl replication - upserted batch: number_upserted=1 batch_size=136
agent.server.replication.acl.policy: acl replication - finished updates
agent.server.replication.acl.policy: ACL replication completed through remote index: index=183
Primary:
consul acl policy update --name test_policy --rules @test.hcl
ID: 5c62c300-a08c-ac54-e00e-8fe8ad5c7503
Name: test_policy
Description:
Datacenters:
Rules:
#testing policy
node_prefix "test-23" {
policy = "write"
}

DC1:
consul acl policy read --name test_policy
ID: 5c62c300-a08c-ac54-e00e-8fe8ad5c7503
Name: test_policy
Description:
Datacenters:
Rules:
#testing policy
node_prefix "test-23" {
policy = "write"
}
curl -k https://localhost:8501/v1/acl/replication?pretty
{
"Enabled": true,
"Running": true,
"SourceDatacenter": "dc",
"ReplicationType": "tokens",
"ReplicatedIndex": 183,
"ReplicatedRoleIndex": 1,
"ReplicatedTokenIndex": 7,
"LastSuccess": "2024-11-20T14:35:40Z",
"LastError": "0001-01-01T00:00:00Z",
"LastErrorMessage": ""
}
DC2:
consul acl policy read --name test_policy
ID: 5c62c300-a08c-ac54-e00e-8fe8ad5c7503
Name: test_policy
Description:
Datacenters:
Rules:
#testing policy
node_prefix "test-22" {
policy = "write"
}
curl -k https://localhost:8501/v1/acl/replication?pretty
{
"Enabled": true,
"Running": true,
"SourceDatacenter": "dc",
"ReplicationType": "tokens",
"ReplicatedIndex": 183,
"ReplicatedRoleIndex": 1,
"ReplicatedTokenIndex": 7,
"LastSuccess": "2024-11-20T14:35:47Z",
"LastError": "0001-01-01T00:00:00Z",
"LastErrorMessage": ""
}
```

Start by reproducing the issue with three WAN-connected clusters, repeatedly creating and updating an ACL policy, then compare the policy contents and replication status on each secondary. Inspect the ACL replication logs and status fields, especially ReplicatedIndex, LastSuccess, and LastErrorMessage. Done means policy updates reliably converge on every secondary and replication status does not report success while a policy is stale.

Written by the indexing model from the issue text.