[Unstable Tests] Unstable tests on Master

Here are 2 unstable tests I found while doing PR https://github.com/hashicorp/consul/pull/8685 from master 86e7274a70db1803f3c4bb68d8dccbc01dbf3b5b

1. [TestACLTokenReap_Primary](https://app.circleci.com/pipelines/github/hashicorp/consul/12566/workflows/af21fc88-41f2-4e0a-96b3-dc823c8645ed/jobs/237174)

```
Failed
=== RUN TestACLTokenReap_Primary/local/one_should_be_reaped
=== CONT TestACLTokenReap_Primary/local/one_should_be_reaped
acl_token_exp_test.go:191:
Error Trace: acl_token_exp_test.go:191
Error: Not equal:
expected: 1
actual : 0
Test: TestACLTokenReap_Primary/local/one_should_be_reaped
--- FAIL: TestACLTokenReap_Primary/local/one_should_be_reaped (0.00s)
```

and

```
Failed
=== RUN TestACLTokenReap_Primary/local/two_should_be_reaped
=== CONT TestACLTokenReap_Primary/local/two_should_be_reaped
acl_token_exp_test.go:208:
Error Trace: acl_token_exp_test.go:208
Error: Not equal:
expected: 1
actual : 0
Test: TestACLTokenReap_Primary/local/two_should_be_reaped
--- FAIL: TestACLTokenReap_Primary/local/two_should_be_reaped (0.00s)
```

2. [TestLeader_SecondaryCA_IntermediateRenew](https://app.circleci.com/pipelines/github/hashicorp/consul/12567/workflows/c58e6f91-5ec5-40c5-9e88-8fcb9266b7e2/jobs/237202)

```
=== RUN TestLeader_SecondaryCA_IntermediateRenew
leader_connect_test.go:302:
Error Trace: leader_connect_test.go:302
Error: Received unexpected error:
x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "sec-13d5osd.consul.ca.813bb1b6.consul")
Test: TestLeader_SecondaryCA_IntermediateRenew
2020-09-15T10:23:50.570Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:015d58ec-f3ab-7862-826f-af14c94e92ee Address:127.0.0.1:26534}]"
2020-09-15T10:23:50.570Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.raft: entering follower state: follower="Node at 127.0.0.1:26534 [Follower]" leader=
2020-09-15T10:23:50.570Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.serf.wan: serf: EventMemberJoin: TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2 127.0.0.1
2020-09-15T10:23:50.571Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.serf.lan: serf: EventMemberJoin: TestLeader_SecondaryCA_IntermediateRenew-node-12 127.0.0.1
2020-09-15T10:23:50.571Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.router: Initializing LAN area manager
2020-09-15T10:23:50.571Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: Adding LAN server: server="TestLeader_SecondaryCA_IntermediateRenew-node-12 (Addr: tcp/127.0.0.1:26534) (DC: dc2)"
2020-09-15T10:23:50.571Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: Handled event for server in area: event=member-join server=TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2 area=wan
2020-09-15T10:23:50.571Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.memberlist.wan: memberlist: Initiating push/pull sync with: TestLeader_SecondaryCA_IntermediateRenew-node-11.dc1 127.0.0.1:26533
2020-09-15T10:23:50.572Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.serf.wan: serf: EventMemberJoin: TestLeader_SecondaryCA_IntermediateRenew-node-11.dc1 127.0.0.1
2020-09-15T10:23:50.572Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: Handled event for server in area: event=member-join server=TestLeader_SecondaryCA_IntermediateRenew-node-11.dc1 area=wan
2020-09-15T10:23:50.671Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.serf.wan: serf: messageJoinType: TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2
2020-09-15T10:23:50.690Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.serf.wan: serf: messageJoinType: TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2
2020-09-15T10:23:50.790Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.serf.wan: serf: messageJoinType: TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2
2020-09-15T10:23:50.790Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.serf.wan: serf: messageJoinType: TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2
2020-09-15T10:23:50.862Z [WARN] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.raft: heartbeat timeout reached, starting election: last-leader=
2020-09-15T10:23:50.862Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.raft: entering candidate state: node="Node at 127.0.0.1:26534 [Candidate]" term=2
2020-09-15T10:23:50.863Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.raft: votes: needed=1
2020-09-15T10:23:50.863Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.raft: vote granted: from=015d58ec-f3ab-7862-826f-af14c94e92ee term=2 tally=1
2020-09-15T10:23:50.863Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.raft: election won: tally=1
2020-09-15T10:23:50.863Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.raft: entering leader state: leader="Node at 127.0.0.1:26534 [Leader]"
2020-09-15T10:23:50.863Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: cluster leadership acquired
2020-09-15T10:23:50.863Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: New leader elected: payload=TestLeader_SecondaryCA_IntermediateRenew-node-12
2020-09-15T10:23:50.863Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=0 found=true
2020-09-15T10:23:50.865Z [DEBUG] connect.ca.consul: consul CA provider configured: id=ad:4a:c6:ab:ef:63:c9:60:1a:51:7f:19:62:e3:e9:d9:0e:76:55:10:6e:74:24:69:28:a1:6c:b8:b9:8f:fd:89 is_primary=false
2020-09-15T10:23:50.868Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: received new intermediate certificate from primary datacenter
2020-09-15T10:23:50.868Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: updated root certificates from primary datacenter
2020-09-15T10:23:50.868Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: initialized secondary datacenter CA with provider: provider=consul
2020-09-15T10:23:50.868Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: started routine: routine="config entry replication"
2020-09-15T10:23:50.868Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: started routine: routine="federation state replication"
2020-09-15T10:23:50.868Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: started routine: routine="federation state anti-entropy"
2020-09-15T10:23:50.869Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: started routine: routine="secondary CA roots watch"
2020-09-15T10:23:50.869Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: started routine: routine="intention replication"
2020-09-15T10:23:50.869Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: started routine: routine="secondary cert renew watch"
2020-09-15T10:23:50.869Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: started routine: routine="CA root pruning"
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: starting Connect intention replication from primary datacenter: primary=dc1
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: Skipping self join check for node since the cluster is too small: node=TestLeader_SecondaryCA_IntermediateRenew-node-12
2020-09-15T10:23:50.869Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: member joined, marking health alive: member=TestLeader_SecondaryCA_IntermediateRenew-node-12
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.replication.config_entry: finished fetching config entries: amount=0
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.replication.config_entry: Config Entry replication: local=0 remote=0
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.replication.config_entry: Config Entry replication: deletions=0 updates=0
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.replication.config_entry: replication completed through remote index: index=1
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: finished fetching remote objects: amount=1
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: diffing replication state: local_amount=0 remote_amount=1
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: diffed replication state: deletions=0 updates=1
2020-09-15T10:23:50.869Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: performing updates: updates=1
2020-09-15T10:23:50.869Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: federation state anti-entropy synced
2020-09-15T10:23:50.870Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: finished updates
2020-09-15T10:23:50.870Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.replication.federation_state: replication completed through remote index: index=10
2020-09-15T10:23:50.870Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: finished fetching remote objects: amount=2
2020-09-15T10:23:50.870Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: diffing replication state: local_amount=1 remote_amount=2
2020-09-15T10:23:50.870Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: diffed replication state: deletions=0 updates=1
2020-09-15T10:23:50.870Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: performing updates: updates=1
2020-09-15T10:23:50.871Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: federation state anti-entropy synced
2020-09-15T10:23:50.871Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: finished updates
2020-09-15T10:23:50.871Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.replication.federation_state: replication completed through remote index: index=14
2020-09-15T10:23:50.872Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: finished fetching remote objects: amount=2
2020-09-15T10:23:50.872Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: diffing replication state: local_amount=2 remote_amount=2
2020-09-15T10:23:50.872Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: diffed replication state: deletions=0 updates=1
2020-09-15T10:23:50.872Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: performing updates: updates=1
2020-09-15T10:23:50.872Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: finished updates
2020-09-15T10:23:50.872Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.replication.federation_state: replication completed through remote index: index=15
2020-09-15T10:23:50.874Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: received new intermediate certificate from primary datacenter
2020-09-15T10:23:50.875Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: updated root certificates from primary datacenter
2020-09-15T10:23:50.879Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: received new intermediate certificate from primary datacenter
2020-09-15T10:23:50.880Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: updated root certificates from primary datacenter
2020-09-15T10:23:50.884Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: received new intermediate certificate from primary datacenter
2020-09-15T10:23:50.885Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: updated root certificates from primary datacenter
2020-09-15T10:23:50.889Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: received new intermediate certificate from primary datacenter
2020-09-15T10:23:50.889Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: updated root certificates from primary datacenter
2020-09-15T10:23:50.893Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: received new intermediate certificate from primary datacenter
2020-09-15T10:23:50.894Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: updated root certificates from primary datacenter
2020-09-15T10:23:50.898Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: received new intermediate certificate from primary datacenter
2020-09-15T10:23:50.899Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: updated root certificates from primary datacenter
2020-09-15T10:23:50.903Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: received new intermediate certificate from primary datacenter
2020-09-15T10:23:50.903Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: updated root certificates from primary datacenter
2020-09-15T10:23:50.908Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: received new intermediate certificate from primary datacenter
2020-09-15T10:23:50.908Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: updated root certificates from primary datacenter
2020-09-15T10:23:50.912Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: received new intermediate certificate from primary datacenter
2020-09-15T10:23:50.914Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: updated root certificates from primary datacenter
2020-09-15T10:23:50.917Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: shutting down server
2020-09-15T10:23:50.917Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopping routine: routine="config entry replication"
2020-09-15T10:23:50.917Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopping routine: routine="federation state replication"
2020-09-15T10:23:50.917Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopping routine: routine="federation state anti-entropy"
2020-09-15T10:23:50.917Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopping routine: routine="secondary CA roots watch"
2020-09-15T10:23:50.917Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopping routine: routine="intention replication"
2020-09-15T10:23:50.917Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopping routine: routine="secondary cert renew watch"
2020-09-15T10:23:50.917Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopping routine: routine="CA root pruning"
2020-09-15T10:23:50.917Z [WARN] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.serf.lan: serf: Shutdown without a Leave
2020-09-15T10:23:50.917Z [ERROR] TestLeader_SecondaryCA_IntermediateRenew-node-12.server: error performing anti-entropy sync of federation state: error="context canceled"
2020-09-15T10:23:50.917Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopped routine: routine="CA root pruning"
2020-09-15T10:23:50.917Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopped routine: routine="federation state anti-entropy"
2020-09-15T10:23:50.917Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.usage_metrics: usage metrics reporter shutting down
2020-09-15T10:23:50.917Z [WARN] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.serf.wan: serf: Shutdown without a Leave
2020-09-15T10:23:50.917Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.router.manager: shutting down
2020-09-15T10:23:50.917Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.router.manager: shutting down
2020-09-15T10:23:50.917Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.router.manager: shutting down
2020-09-15T10:23:50.918Z [ERROR] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: error renewing intermediate certs: routine="secondary cert renew watch" error="Failed to set the intermediate certificate with the CA provider: raft is already shutdown"
2020-09-15T10:23:50.918Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopped routine: routine="secondary cert renew watch"
2020-09-15T10:23:50.918Z [ERROR] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.rpc: RPC failed to server in DC: server=127.0.0.1:26531 datacenter=dc1 method=ConnectCA.Roots error="rpc error making call: EOF"
2020-09-15T10:23:50.918Z [ERROR] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.rpc: RPC failed to server in DC: server=127.0.0.1:26531 datacenter=dc1 method=FederationState.List error="rpc error making call: EOF"
2020-09-15T10:23:50.918Z [ERROR] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.rpc: RPC failed to server in DC: server=127.0.0.1:26531 datacenter=dc1 method=ConfigEntry.ListAll error="rpc error making call: EOF"
2020-09-15T10:23:50.918Z [WARN] TestLeader_SecondaryCA_IntermediateRenew-node-12.replication.federation_state: replication error (will retry if still leader): error="failed to retrieve federation states: rpc error making call: EOF"
2020-09-15T10:23:50.918Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.replication.federation_state: stopped replication
2020-09-15T10:23:50.918Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopped routine: routine="federation state replication"
2020-09-15T10:23:50.918Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.replication.config_entry: stopped replication
2020-09-15T10:23:50.918Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopped routine: routine="config entry replication"
2020-09-15T10:23:50.918Z [ERROR] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.rpc: RPC failed to server in DC: server=127.0.0.1:26531 datacenter=dc1 method=Intention.List error="rpc error making call: EOF"
2020-09-15T10:23:50.918Z [ERROR] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: error replicating intentions: routine="intention replication" error="rpc error making call: EOF"
2020-09-15T10:23:50.918Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopped routine: routine="intention replication"
2020-09-15T10:23:50.918Z [ERROR] TestLeader_SecondaryCA_IntermediateRenew-node-12.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: rpc error making call: EOF"
2020-09-15T10:23:50.918Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-12.leader: stopped routine: routine="secondary CA roots watch"
2020-09-15T10:23:50.289Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ee6868b7-0045-50d8-16d7-c946b96d03d4 Address:127.0.0.1:26531}]"
2020-09-15T10:23:50.289Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.raft: entering follower state: follower="Node at 127.0.0.1:26531 [Follower]" leader=
2020-09-15T10:23:50.289Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.serf.wan: serf: EventMemberJoin: TestLeader_SecondaryCA_IntermediateRenew-node-11.dc1 127.0.0.1
2020-09-15T10:23:50.290Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.serf.lan: serf: EventMemberJoin: TestLeader_SecondaryCA_IntermediateRenew-node-11 127.0.0.1
2020-09-15T10:23:50.290Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.router: Initializing LAN area manager
2020-09-15T10:23:50.290Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: Adding LAN server: server="TestLeader_SecondaryCA_IntermediateRenew-node-11 (Addr: tcp/127.0.0.1:26531) (DC: dc1)"
2020-09-15T10:23:50.290Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: Handled event for server in area: event=member-join server=TestLeader_SecondaryCA_IntermediateRenew-node-11.dc1 area=wan
2020-09-15T10:23:50.496Z [WARN] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.raft: heartbeat timeout reached, starting election: last-leader=
2020-09-15T10:23:50.496Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.raft: entering candidate state: node="Node at 127.0.0.1:26531 [Candidate]" term=2
2020-09-15T10:23:50.496Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.raft: votes: needed=1
2020-09-15T10:23:50.496Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.raft: vote granted: from=ee6868b7-0045-50d8-16d7-c946b96d03d4 term=2 tally=1
2020-09-15T10:23:50.496Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.raft: election won: tally=1
2020-09-15T10:23:50.496Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.raft: entering leader state: leader="Node at 127.0.0.1:26531 [Leader]"
2020-09-15T10:23:50.496Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: cluster leadership acquired
2020-09-15T10:23:50.496Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: New leader elected: payload=TestLeader_SecondaryCA_IntermediateRenew-node-11
2020-09-15T10:23:50.497Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26531
2020-09-15T10:23:50.498Z [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true
2020-09-15T10:23:50.499Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.connect: initialized primary datacenter CA with provider: provider=consul
2020-09-15T10:23:50.499Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.leader: started routine: routine="federation state anti-entropy"
2020-09-15T10:23:50.499Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.leader: started routine: routine="federation state pruning"
2020-09-15T10:23:50.499Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.leader: started routine: routine="CA root pruning"
2020-09-15T10:23:50.499Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: Skipping self join check for node since the cluster is too small: node=TestLeader_SecondaryCA_IntermediateRenew-node-11
2020-09-15T10:23:50.499Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: member joined, marking health alive: member=TestLeader_SecondaryCA_IntermediateRenew-node-11
2020-09-15T10:23:50.500Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: federation state anti-entropy synced
2020-09-15T10:23:50.571Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:53734
2020-09-15T10:23:50.572Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.serf.wan: serf: EventMemberJoin: TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2 127.0.0.1
2020-09-15T10:23:50.572Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: Handled event for server in area: event=member-join server=TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2 area=wan
2020-09-15T10:23:50.590Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.serf.wan: serf: messageJoinType: TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2
2020-09-15T10:23:50.671Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.serf.wan: serf: messageJoinType: TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2
2020-09-15T10:23:50.671Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.serf.wan: serf: messageJoinType: TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2
2020-09-15T10:23:50.771Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.serf.wan: serf: messageJoinType: TestLeader_SecondaryCA_IntermediateRenew-node-12.dc2
2020-09-15T10:23:50.797Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: Skipping self join check for node since the cluster is too small: node=TestLeader_SecondaryCA_IntermediateRenew-node-11
2020-09-15T10:23:50.919Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: shutting down server
2020-09-15T10:23:50.919Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.leader: stopping routine: routine="federation state anti-entropy"
2020-09-15T10:23:50.919Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.leader: stopping routine: routine="federation state pruning"
2020-09-15T10:23:50.919Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.leader: stopping routine: routine="CA root pruning"
2020-09-15T10:23:50.919Z [WARN] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.serf.lan: serf: Shutdown without a Leave
2020-09-15T10:23:50.919Z [ERROR] TestLeader_SecondaryCA_IntermediateRenew-node-11.server: error performing anti-entropy sync of federation state: error="context canceled"
2020-09-15T10:23:50.919Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.leader: stopped routine: routine="federation state pruning"
2020-09-15T10:23:50.919Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.leader: stopped routine: routine="federation state anti-entropy"
2020-09-15T10:23:50.919Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.usage_metrics: usage metrics reporter shutting down
2020-09-15T10:23:50.919Z [DEBUG] TestLeader_SecondaryCA_IntermediateRenew-node-11.leader: stopped routine: routine="CA root pruning"
2020-09-15T10:23:50.920Z [WARN] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.serf.wan: serf: Shutdown without a Leave
2020-09-15T10:23:50.920Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.router.manager: shutting down
2020-09-15T10:23:50.920Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.router.manager: shutting down
2020-09-15T10:23:50.920Z [INFO] TestLeader_SecondaryCA_IntermediateRenew-node-11.server.router.manager: shutting down
--- FAIL: TestLeader_SecondaryCA_IntermediateRenew (0.64s)
```