Networking issue with raft after leader election

#### Overview of the Issue

I have three server instances running as containers on three different nodes in a docker swarm mode. Since the server containers will need to address each other individually, I need to use endpoint mode `dnsrr` and `max_replicas_per_node` set to 1. For the TLS keys, I created two secrets `consul_ca` and `consul_ca_key`.

This setup seems to work fine for the bootstrapping process including the leader election. However, for synchronizing the server states, raft seems to run into some network issue. A typical error from the server logs would look like this:

```
[ERROR] agent.server.raft: failed to appendEntries to: peer="{Voter 32be4381-f025-cb2b-852d-2a721b1a2938 10.0.1.3:8300}" error="read tcp 10.0.1.4:36031->10.0.1.3:8300: i/o timeout"
```

#### Reproduction Steps

1. Create three nodes in docker swarm mode and log into one of the manager nodes
1. Run `mkdir ca; sudo docker run --rm -v ./ca:/ca consul sh -c "cd /ca; consul tls ca create -domain example.com"` (please replace `example.com` by your domain)
1. Run `sudo cat ./example.com-agent-ca-key.pem | sudo docker secret create consul_ca_key -` (please replace `example.com` by your domain)
1. Run `sudo cat ./example.com-agent-ca.pem | sudo docker secret create consul_ca -` (please replace `example.com` by your domain)
1. Run `sudo docker run --rm consul consul keygen` and copy the gossip key for later use
1. Create a `./server.hcl` containing the following server configuration (please replace `GOSSIP_ENCRYPTION_KEY` by your gossip key generated in the previous step):
```
server = true
datacenter = "dc1"

bind_addr = "{{ GetInterfaceIP \"eth0\" }}"

data_dir = "/consul/data"
encrypt = "GOSSIP_ENCRYPTION_KEY"

ports {
dns = -1
http = -1
https = -1
grpc = -1
server = 8300
serf_lan = 8301
serf_wan = -1
}

tls {
defaults {
ca_file = "/run/secrets/consul_ca"
cert_file = "/dc1-server-consul-0.pem"
key_file = "/dc1-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
}
internal_rpc {
verify_server_hostname = false
}
}

auto_encrypt {
allow_tls = true
}

retry_join = ["consul_server"]
```

7. Create a docker compose file `./docker-compose.yml`:
```
version: '3.9'

services:
server:
image: consul:1.13
command: /bin/sh -c "consul tls cert create -server -ca=/run/secrets/consul_ca -key=/run/secrets/consul_ca_key; consul agent -bootstrap-expect 3 -config-file=/etc/consul.d/server.hcl -log-level=TRACE"
deploy:
replicas: 3
placement:
max_replicas_per_node: 1
endpoint_mode: dnsrr
ports:
- target: 8300
published: 8300
protocol: tcp
mode: host
- target: 8301
published: 8301
protocol: tcp
mode: host
- target: 8301
published: 8301
protocol: udp
mode: host
configs:
- source: server
target: /etc/consul.d/server.hcl
secrets:
- consul_ca
- consul_ca_key
networks:
- consul
volumes:
- data:/consul/data

configs:
server:
file: ./server.hcl

secrets:
consul_ca:
external: true
consul_ca_key:
external: true

networks:
consul:
attachable: true

volumes:
data:
```
8. Run `sudo docker stack deploy --compose-file ./docker-compose.yml consul`
9. Check the logs by running `sudo docker service logs consul_server`

### Consul info for both Client and Server

Client info
There is no client

Server info

Result of running `sudo docker ps` to obtain the container id, `sudo docker exec -it CONTAINERID /bin/sh` (replacing `CONTAINERID` by the appropriate value) and finally `consul info` within the container:
```
Error querying agent: Get "http://127.0.0.1:8500/v1/agent/self": dial tcp 127.0.0.1:8500: connect: connection refused
```

### Operating system and Environment details

See reproduction steps. It is the official hashicorp consul docker image 1.13.3 deployed to docker swarm mode.

### Log Fragments

On the leader the log repeats a pattern simmilar to this (repeated identical lines were removed):
```
2022-10-21T18:33:00.387Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2022-10-21T18:33:00.455Z [TRACE] agent.server.usage_metrics: Starting usage run
2022-10-21T18:33:01.067Z [TRACE] agent.server: rpc_server_call: method=Coordinate.Update errored=false request_type=write rpc_type=net/rpc leader=true target_datacenter=dc1 locality=local
2022-10-21T18:33:01.465Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2022-10-21T18:33:02.380Z [DEBUG] agent.server.memberlist.lan: memberlist: Stream connection from=10.0.1.2:47598
2022-10-21T18:33:02.387Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2022-10-21T18:33:02.611Z [TRACE] agent.server: rpc_server_call: method=Coordinate.Update errored=false request_type=write rpc_type=net/rpc leader=true target_datacenter=dc1 locality=local
2022-10-21T18:33:03.464Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2022-10-21T18:33:09.456Z [TRACE] agent.server.usage_metrics: Starting usage run
2022-10-21T18:33:09.464Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2022-10-21T18:33:10.500Z [TRACE] agent.tlsutil: OutgoingRPCConfig: version=3
2022-10-21T18:33:11.464Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2022-10-21T18:33:14.342Z [TRACE] agent.server: rpc_server_call: method=Coordinate.Update errored=false request_type=write rpc_type=net/rpc leader=true target_datacenter=dc1 locality=local
2022-10-21T18:33:14.386Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2022-10-21T18:33:18.456Z [TRACE] agent.server.usage_metrics: Starting usage run
2022-10-21T18:33:19.316Z [DEBUG] agent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 576d6aba27fc 10.0.1.3:8301
2022-10-21T18:33:19.464Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2022-10-21T18:33:20.406Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Voter dbbe3bf7-74d6-babb-f47c-98985b95ffcf 10.0.1.2:8300}" error="read tcp 10.0.1.4:53515->10.0.1.2:8300: i/o timeout"
2022-10-21T18:33:20.488Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Voter 32be4381-f025-cb2b-852d-2a721b1a2938 10.0.1.3:8300}" error="read tcp 10.0.1.4:36031->10.0.1.3:8300: i/o timeout"
```

Link to the full logs:
[Hashicorp/consul | Networking issue with raft after leader election | Full cluster leader log](https://gist.github.com/alphanoob1337/807eb1f6d558c6f234fdf0156b96a829)
[Hashicorp/consul | Networking issue with raft after leader election | Full cluster follower log 1](https://gist.github.com/alphanoob1337/06002f8434005449245beb5fd9fc15ab)
[Hashicorp/consul | Networking issue with raft after leader election | Full cluster follower log 2](https://gist.github.com/alphanoob1337/870f3415e1426e1fe04ab071b83e7ea8)