agent.server.raft: failed to contact: server-id, leader gets replaced

#### Overview of the Issue

We have deployed the consul cluster in HA mode to AWS EC2 instances(5 nodes). We notice from time to time and it is very inconsistent that the leader changes for the cluster. The consul logs are provided below for when this happens.

#### Reproduction Steps

not consistent so not reproducible

### Consul info for both Client and Server

Server info

```
/usr/local/bin/consul info
gent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = a9322b9c
version = 1.8.3
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = omited:8300
server = true
raft:
applied_index = 12817245
commit_index = 12817245
fsm_pending = 0
last_contact = 0
last_log_index = 12817245
last_log_term = 23
last_snapshot_index = 12817072
last_snapshot_term = 23
latest_configuration = [omitted]
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 23
runtime:
arch = amd64
cpu_count = 2
goroutines = 200
max_procs = 2
os = linux
version = go1.14.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 13
failed = 0
health_score = 0
intent_queue = 0
left = 3
member_time = 74003
members = 11
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 22154
members = 5
query_queue = 0
query_time = 1
```

### Operating system and Environment details

```
4.14.214-160.339.amzn2.x86_64 #1 SMP Sun Jan 10 05:53:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
```
### Log Fragments

```
March 10th 2021, 16:10:32.559 agent.server.raft: failed to contact: server-id=d8c4b13d-5a3c-b203-2d33-ac46884478c8 time=1.202618808s warning consul
March 10th 2021, 16:10:32.559 agent.server.raft: failed to contact: server-id=a0eca217-5682-80a3-ba4e-f32862845139 time=1.174153532s warning consul
March 10th 2021, 16:10:32.559 agent.server.raft: failed to contact: server-id=0f17a602-6d0e-0bb9-46fe-1f36ea09fd1f time=1.174704467s warning consul
March 10th 2021, 16:10:32.560 agent.server.raft: aborting pipeline replication: peer="{Voter 0f17a602-6d0e-0bb9-46fe-1f36ea09fd1f ip-1:8300}" notice consul
March 10th 2021, 16:10:32.560 agent.server.raft: aborting pipeline replication: peer="{Voter d8c4b13d-5a3c-b203-2d33-ac46884478c8 ip-4:8300}" notice consul
March 10th 2021, 16:10:32.560 agent.server.raft: failed to contact: server-id=16edce45-197c-459d-ac1a-461d2a02be23 time=1.253019255s warning consul
March 10th 2021, 16:10:32.560 agent.server.raft: failed to contact quorum of nodes, stepping down warning consul
March 10th 2021, 16:10:32.560 agent.server.raft: entering follower state: follower="Node at ip-5:8300 [Follower]" leader= notice consul
March 10th 2021, 16:10:32.561 agent.server: cluster leadership lost notice consul
March 10th 2021, 16:10:32.561 agent.server.raft: aborting pipeline replication: peer="{Voter a0eca217-5682-80a3-ba4e-f32862845139 ip-2:8300}" notice consul
March 10th 2021, 16:10:32.561 agent.server.raft: aborting pipeline replication: peer="{Voter 16edce45-197c-459d-ac1a-461d2a02be23 ip-3:8300}" notice consul
March 10th 2021, 16:10:33.754 agent.server.raft: heartbeat timeout reached, starting election: last-leader=ip-5:8300 warning consul
March 10th 2021, 16:10:33.754 agent.server.raft: entering candidate state: node="Node at ip-4:8300 [Candidate]" term=168 notice consul
March 10th 2021, 16:10:33.756 agent.server.raft: rejecting vote request since we have a leader: from=ip-4:8300 leader=ip-5:8300 warning consul
March 10th 2021, 16:10:33.756 agent.server.raft: rejecting vote request since we have a leader: from=ip-4:8300 leader=ip-5:8300 warning consul ip-4
March 10th 2021, 16:10:33.759 agent.server.raft: rejecting vote request since we have a leader: from=ip-4:8300 leader=ip-5:8300 warning consul ip-5
March 10th 2021, 16:10:33.818 agent.server.raft: heartbeat timeout reached, starting election: last-leader=ip-5:8300 warning consul
March 10th 2021, 16:10:33.819 agent.server.raft: entering candidate state: node="Node at ip-2:8300 [Candidate]" term=168 notice consul
March 10th 2021, 16:10:33.824 agent.server.raft: rejecting vote request since we have a leader: from=ip-2:8300 leader=ip-5:8300 warning consul ip-5
March 10th 2021, 16:10:33.824 agent.server.raft: rejecting vote request since we have a leader: from=ip-2:8300 leader=ip-5:8300 warning consul ip-4
March 10th 2021, 16:10:33.824 agent.server.raft: duplicate requestVote for same term: term=168 notice consul
March 10th 2021, 16:10:33.826 agent.server.raft: duplicate requestVote for same term: term=168 notice consul
March 10th 2021, 16:10:34.583 agent.server.raft: heartbeat timeout reached, starting election: last-leader=ip-5:8300 warning consul ip-5
March 10th 2021, 16:10:34.583 agent.server.raft: entering candidate state: node="Node at ip-3:8300 [Candidate]" term=168 notice consul ip-5
March 10th 2021, 16:10:34.586 agent.server.raft: rejecting vote request since we have a leader: from=ip-3:8300 leader=ip-5:8300 warning consul ip-4
March 10th 2021, 16:10:34.586 agent.server.raft: duplicate requestVote for same term: term=168 notice consul
March 10th 2021, 16:10:34.590 agent.server.raft: duplicate requestVote for same term: term=168 notice consul
March 10th 2021, 16:10:34.593 agent.server.raft: duplicate requestVote for same term: term=168 notice consul
March 10th 2021, 16:10:34.648 agent.server.raft: heartbeat timeout reached, starting election: last-leader=ip-5:8300 warning consul ip-4
March 10th 2021, 16:10:34.649 agent.server.raft: entering candidate state: node="Node at ip-1:8300 [Candidate]" term=168 notice consul ip-4
March 10th 2021, 16:10:34.651 agent.server.raft: duplicate requestVote for same term: term=168 notice consul
March 10th 2021, 16:10:34.653 agent.server.raft: duplicate requestVote for same term: term=168 notice consul
March 10th 2021, 16:10:34.653 agent.server.raft: duplicate requestVote for same term: term=168 notice consul ip-5
March 10th 2021, 16:10:34.655 agent.server.raft: duplicate requestVote for same term: term=168 notice consul
March 10th 2021, 16:10:34.847 agent.server.raft: entering candidate state: node="Node at ip-2:8300 [Candidate]" term=169 notice consul
March 10th 2021, 16:10:34.847 agent.server.raft: Election timeout reached, restarting election warning consul
March 10th 2021, 16:10:34.855 agent.server.raft: entering leader state: leader="Node at ip-2:8300 [Leader]" notice consul
March 10th 2021, 16:10:34.855 agent.server.raft: entering follower state: follower="Node at ip-1:8300 [Follower]" leader= notice consul ip-4
March 10th 2021, 16:10:34.855 agent.server.raft: election won: tally=3 notice consul
March 10th 2021, 16:10:34.856 agent.server.raft: added peer, starting replication: peer=0f17a602-6d0e-0bb9-46fe-1f36ea09fd1f notice consul
March 10th 2021, 16:10:34.856 agent.server.raft: added peer, starting replication: peer=d8c4b13d-5a3c-b203-2d33-ac46884478c8 notice consul
March 10th 2021, 16:10:34.856 agent.server: cluster leadership acquired notice consul
March 10th 2021, 16:10:34.856 agent.server.raft: added peer, starting replication: peer=f56312fc-6fb3-f698-563d-6dff30bf67cb notice consul
March 10th 2021, 16:10:34.856 agent.server.raft: entering follower state: follower="Node at ip-3:8300 [Follower]" leader= notice consul ip-5
March 10th 2021, 16:10:34.856 agent.server: New leader elected: payload= notice consul
March 10th 2021, 16:10:34.856 agent.server.raft: added peer, starting replication: peer=16edce45-197c-459d-ac1a-461d2a02be23 notice consul
March 10th 2021, 16:10:34.857 agent.server.raft: entering follower state: follower="Node at ip-4:8300 [Follower]" leader= notice consul
March 10th 2021, 16:10:34.857 agent.server.raft: pipelining replication: peer="{Voter f56312fc-6fb3-f698-563d-6dff30bf67cb ip-5:8300}" notice consul
March 10th 2021, 16:10:34.857 agent.server.raft: pipelining replication: peer="{Voter 0f17a602-6d0e-0bb9-46fe-1f36ea09fd1f ip-1:8300}" notice consul
March 10th 2021, 16:10:34.860 agent.server.raft: pipelining replication: peer="{Voter d8c4b13d-5a3c-b203-2d33-ac46884478c8 ip-4:8300}" notice consul
March 10th 2021, 16:10:34.861 agent.server.raft: pipelining replication: peer="{Voter 16edce45-197c-459d-ac1a-461d2a02be23 ip-3:8300}" notice consul
March 10th 2021, 16:10:34.863 agent.server: initializing acls notice consul
March 10th 2021, 16:10:34.864 agent.leader: started routine: routine="federation state anti-entropy" notice consul
March 10th 2021, 16:10:34.864 agent.leader: started routine: routine="legacy ACL token upgrade" notice consul
March 10th 2021, 16:10:34.864 agent.leader: started routine: routine="acl token reaping" notice consul
March 10th 2021, 16:10:34.865 agent.leader: started routine: routine="federation state pruning" notice consul
March 10th 2021, 16:10:34.865 agent.leader: started routine: routine="CA root pruning" notice consul
March 10th 2021, 16:10:34.974 agent.server: New leader elected: payload= notice consul ip-4
March 10th 2021, 16:10:34.979 agent.server: New leader elected: payload= notice consul
March 10th 2021, 16:10:35.002 agent.server: New leader elected: payload= notice consul ip-5
March 10th 2021, 16:10:35.072 agent.server: New leader elected: payload= notice consul
March 10th 2021, 16:10:35.782 agent.server.coordinate: Batch update failed: error="node is not the leader" warning consul
March 10th 2021, 16:14:56.614 agent.server: error performing anti-entropy sync of federation state: error="context canceled" err consul
```
We cannot see exactly what's causing and how we can mitigate this? our expectation is that leader should not be changing apart of time when we do the deploy. The deployment strategy we use is a/b

Thanks in advance for your help :)