Consul write api hangs for half hour before return rpc error

version: 0.8.4

when this happens, the leader repetively print:

> 2017/12/05 11:10:21 [WARN] raft: AppendEntries to {Voter 10.0.0.9:8300 10.0.0.9:8300} rejected, sending older logs (next: 14336)
> 2017/12/05 11:10:21 [ERR] raft: Failed to get log at index 14335: log not found

and the follower prints:

> 2017/12/05 11:10:11 [ERR] consul: failed to reconcile member: {server-1 10.0.0.10 8301 map[build:0.8.4:f436077 vsn_max:3 raft_vsn:2 wan_join_port:8302 dc:dc1 id:c29aeab8-3731-17d0-8796-b7af5d3953ea port:8300 role:consul vsn:2 vsn_min:2] alive 1 5 2 2 5 4}: leadership lost while committing log
> 2017/12/05 11:10:11 [ERR] consul: failed to reconcile: leadership lost while committing log
> 2017/12/05 11:10:11 [INFO] consul: cluster leadership lost
> 2017/12/05 11:10:18 [WARN] raft: Previous log term mis-match: ours: 4 remote: 5
> 2017/12/05 11:10:18 [INFO] consul: New leader elected: server-1
> 2017/12/05 11:10:18 [INFO] snapshot: Creating new snapshot at /opt/application/consul-works/data-dir/raft/snapshots/5-14336-1512472218803.tmp
>
> 2017/12/05 11:10:18 [INFO] snapshot: reaping snapshot /opt/application/consul-works/data-dir/raft/snapshots/3-12783-1512459076524
> 2017/12/05 11:10:18 [INFO] raft: Copied 625724 bytes to local snapshot
> 2017/12/05 11:10:18 [INFO] raft: Compacting logs from 14333 to 4113
> 2017/12/05 11:10:18 [INFO] raft: Installed remote snapshot
> 2017/12/05 11:10:18 [WARN] raft: Previous log term mis-match: ours: 4 remote: 5
> 2017/12/05 11:10:18 [INFO] snapshot: Creating new snapshot at /opt/application/consul-works/data-dir/raft/snapshots/5-14336-1512472218916.tmp