hashicorp / hashicorp/consul

Deflake agent/consul.TestAutopilot_CleanupDeadServerPeriodic

Open
#7,084 1 comment 0 reactions 0 assignees View on GitHub
theme/testing
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

Ran into an errant failure today -- stubbing this out with logs

related #7042

```
=== RUN TestAutopilot_CleanupDeadServer
=== PAUSE TestAutopilot_CleanupDeadServer
=== CONT TestAutopilot_CleanupDeadServer
--- FAIL: TestAutopilot_CleanupDeadServer (2.63s)
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:46.354622 [DEBUG] tlsutil: Update with version 1
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:46.354958 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Initial configuration (index=0): []
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Node at 127.0.0.1:16654 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:46.526042 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-236.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:46.539995 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-236 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:46.540173 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServer-node-236 (Addr: tcp/127.0.0.1:16654) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:46.540271 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServer-node-236.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:46.546980 [DEBUG] tlsutil: Update with version 1
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:46.547536 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Initial configuration (index=0): []
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Node at 127.0.0.1:16669 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:46.716496 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-241.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:46.724362 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-241 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:46.724643 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServer-node-241.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:46.724651 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServer-node-241 (Addr: tcp/127.0.0.1:16669) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:46.736739 [DEBUG] tlsutil: Update with version 1
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:46.737050 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
writer.go:21: 2020/01/17 10:56:46 [WARN] raft: no known peers, aborting election
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Initial configuration (index=0): []
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Node at 127.0.0.1:16681 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:46.984952 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-245.dc1 127.0.0.1
writer.go:21: 2020/01/17 10:56:46 [WARN] raft: no known peers, aborting election
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:46.997734 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-245 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:46.997910 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServer-node-245.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:46.997919 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServer-node-245 (Addr: tcp/127.0.0.1:16681) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServer-node-248 - 2020/01/17 10:56:47.005292 [DEBUG] tlsutil: Update with version 1
log.go:172: TestAutopilot_CleanupDeadServer-node-248 - 2020/01/17 10:56:47.005898 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: Initial configuration (index=0): []
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: Node at 127.0.0.1:16690 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServer-node-248 - 2020/01/17 10:56:47.274824 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-248.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-248 - 2020/01/17 10:56:47.287250 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-248 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-248 - 2020/01/17 10:56:47.287330 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServer-node-248 (Addr: tcp/127.0.0.1:16690) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServer-node-248 - 2020/01/17 10:56:47.287370 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServer-node-248.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServer-node-251 - 2020/01/17 10:56:47.295258 [DEBUG] tlsutil: Update with version 1
log.go:172: TestAutopilot_CleanupDeadServer-node-251 - 2020/01/17 10:56:47.296333 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
writer.go:21: 2020/01/17 10:56:47 [WARN] raft: no known peers, aborting election
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: Initial configuration (index=0): []
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: Node at 127.0.0.1:16699 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServer-node-251 - 2020/01/17 10:56:47.564140 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-251.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-251 - 2020/01/17 10:56:47.571916 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-251 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-251 - 2020/01/17 10:56:47.572066 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServer-node-251 (Addr: tcp/127.0.0.1:16699) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServer-node-251 - 2020/01/17 10:56:47.572077 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServer-node-251.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.573041 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:16655
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.573116 [DEBUG] memberlist: Stream connection from=127.0.0.1:64599
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.573567 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-241 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.573702 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServer-node-241 (Addr: tcp/127.0.0.1:16669) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.573954 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-236 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.574095 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServer-node-236 (Addr: tcp/127.0.0.1:16654) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:47.575177 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:16655
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.575184 [ERR] memberlist: Error accepting TCP connection: accept tcp 127.0.0.1:16671: accept: too many open files
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.575177 [ERR] memberlist: Error accepting TCP connection: accept tcp 127.0.0.1:16655: accept: too many open files
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.575233 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:16671
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.575307 [DEBUG] memberlist: Stream connection from=127.0.0.1:64606
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.575369 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:16656
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:47.575406 [DEBUG] memberlist: Failed to join 127.0.0.1: EOF
helper_test.go:128: 1 error occurred:
* Failed to join 127.0.0.1: EOF


log.go:172: TestAutopilot_CleanupDeadServer-node-251 - 2020/01/17 10:56:47.575436 [INFO] consul: shutting down server
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.575483 [DEBUG] memberlist: Failed to join 127.0.0.1: read tcp 127.0.0.1:64604->127.0.0.1:16671: read: connection reset by peer
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.575528 [DEBUG] consul: Failed to flood-join "TestAutopilot_CleanupDeadServer-node-241" at 127.0.0.1:16671: 1 error occurred:
* Failed to join 127.0.0.1: read tcp 127.0.0.1:64604->127.0.0.1:16671: read: connection reset by peer
log.go:172: TestAutopilot_CleanupDeadServer-node-251 - 2020/01/17 10:56:47.575486 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.575839 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-241.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.575937 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServer-node-241.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.575973 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServer-node-236.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.576063 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServer-node-236.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.576064 [DEBUG] consul: Successfully performed flood-join for "TestAutopilot_CleanupDeadServer-node-236" at 127.0.0.1:16656
writer.go:21: 2020/01/17 10:56:47 [WARN] raft: no known peers, aborting election
log.go:172: TestAutopilot_CleanupDeadServer-node-251 - 2020/01/17 10:56:47.605938 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.617621 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241.dc1
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.617888 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241.dc1
log.go:172: TestAutopilot_CleanupDeadServer-node-251 - 2020/01/17 10:56:47.620244 [INFO] manager: shutting down
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.631027 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241.dc1
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.631369 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.631442 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.632084 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241.dc1
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.632242 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241.dc1
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.641818 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.641846 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.641828 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.716938 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241.dc1
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:47.726658 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.726900 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241.dc1
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.740637 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241
log.go:172: TestAutopilot_CleanupDeadServer-node-248 - 2020/01/17 10:56:47.779093 [INFO] consul: shutting down server
log.go:172: TestAutopilot_CleanupDeadServer-node-248 - 2020/01/17 10:56:47.779249 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.817898 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241.dc1
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:47.825989 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServer-node-241
log.go:172: TestAutopilot_CleanupDeadServer-node-248 - 2020/01/17 10:56:47.851215 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServer-node-248 - 2020/01/17 10:56:47.873475 [INFO] manager: shutting down
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:48.064340 [INFO] consul: shutting down server
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:48.064461 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:48.106535 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServer-node-245 - 2020/01/17 10:56:48.148033 [INFO] manager: shutting down
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:48.302428 [INFO] consul: shutting down server
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:48.302491 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:48.368325 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.390677 [DEBUG] memberlist: Failed ping: TestAutopilot_CleanupDeadServer-node-241 (timeout reached)
log.go:172: TestAutopilot_CleanupDeadServer-node-241 - 2020/01/17 10:56:48.402098 [INFO] manager: shutting down
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.440669 [INFO] memberlist: Suspect TestAutopilot_CleanupDeadServer-node-241 has failed, no acks received
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.478476 [DEBUG] memberlist: Failed ping: TestAutopilot_CleanupDeadServer-node-241.dc1 (timeout reached)
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.528301 [INFO] memberlist: Suspect TestAutopilot_CleanupDeadServer-node-241.dc1 has failed, no acks received
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.569147 [INFO] consul: shutting down server
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.569292 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.580265 [DEBUG] memberlist: Failed ping: TestAutopilot_CleanupDeadServer-node-241.dc1 (timeout reached)
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.591689 [DEBUG] memberlist: Failed ping: TestAutopilot_CleanupDeadServer-node-241 (timeout reached)
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.611805 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.641858 [INFO] memberlist: Marking TestAutopilot_CleanupDeadServer-node-241 as failed, suspect timeout reached (0 peer confirmations)
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.641990 [INFO] serf: EventMemberFailed: TestAutopilot_CleanupDeadServer-node-241 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.662135 [INFO] manager: shutting down
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.730058 [INFO] memberlist: Marking TestAutopilot_CleanupDeadServer-node-241.dc1 as failed, suspect timeout reached (0 peer confirmations)
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.730073 [INFO] memberlist: Suspect TestAutopilot_CleanupDeadServer-node-241.dc1 has failed, no acks received
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.730211 [INFO] serf: EventMemberFailed: TestAutopilot_CleanupDeadServer-node-241.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServer-node-236 - 2020/01/17 10:56:48.741703 [INFO] memberlist: Suspect TestAutopilot_CleanupDeadServer-node-241 has failed, no acks received
FAIL agent/consul.TestAutopilot_CleanupDeadServer (2.63s)
=== RUN TestAutopilot_CleanupDeadServerPeriodic
=== PAUSE TestAutopilot_CleanupDeadServerPeriodic
=== CONT TestAutopilot_CleanupDeadServerPeriodic
--- FAIL: TestAutopilot_CleanupDeadServerPeriodic (3.03s)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:46.320952 [DEBUG] tlsutil: Update with version 1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:46.321381 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:b2142876-cec7-836d-168b-42b00fd9c321 Address:127.0.0.1:16651}]
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Node at 127.0.0.1:16651 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:46.532889 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-235.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:46.539995 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-235 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:46.540186 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServerPeriodic-node-235.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:46.540193 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServerPeriodic-node-235 (Addr: tcp/127.0.0.1:16651) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:46.554635 [DEBUG] tlsutil: Update with version 1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:46.555392 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Initial configuration (index=0): []
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Node at 127.0.0.1:16672 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:46.716485 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:46.724382 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-242 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:46.724594 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServerPeriodic-node-242 (Addr: tcp/127.0.0.1:16672) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:46.724609 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:46.744387 [DEBUG] tlsutil: Update with version 1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:46.744788 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
writer.go:21: 2020/01/17 10:56:46 [WARN] raft: Heartbeat timeout from "" reached, starting election
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Node at 127.0.0.1:16651 [Candidate] entering Candidate state in term 2
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Election won. Tally: 1
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Node at 127.0.0.1:16651 [Leader] entering Leader state
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:46.882363 [INFO] consul: cluster leadership acquired
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:46.882440 [INFO] consul: New leader elected: TestAutopilot_CleanupDeadServerPeriodic-node-235
writer.go:21: 2020/01/17 10:56:46 [WARN] raft: no known peers, aborting election
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Initial configuration (index=0): []
writer.go:21: 2020/01/17 10:56:46 [INFO] raft: Node at 127.0.0.1:16684 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:47.005503 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-246.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:47.011991 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-246 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:47.012151 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServerPeriodic-node-246 (Addr: tcp/127.0.0.1:16684) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:47.012166 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServerPeriodic-node-246.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-249 - 2020/01/17 10:56:47.026269 [DEBUG] tlsutil: Update with version 1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-249 - 2020/01/17 10:56:47.026747 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.039488 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.039615 [DEBUG] consul CA provider configured ID=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 IsPrimary=true
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.064721 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-235": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.150924 [INFO] connect: initialized primary datacenter CA with provider "consul"
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.150996 [INFO] leader: started CA root pruning routine
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.151179 [DEBUG] consul: Skipping self join check for "TestAutopilot_CleanupDeadServerPeriodic-node-235" since the cluster is too small
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.151252 [INFO] consul: member 'TestAutopilot_CleanupDeadServerPeriodic-node-235' joined, marking health alive
writer.go:21: 2020/01/17 10:56:47 [WARN] raft: no known peers, aborting election
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.266817 [DEBUG] consul: Skipping self join check for "TestAutopilot_CleanupDeadServerPeriodic-node-235" since the cluster is too small
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: Initial configuration (index=0): []
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: Node at 127.0.0.1:16693 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-249 - 2020/01/17 10:56:47.294836 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-249.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-249 - 2020/01/17 10:56:47.305311 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-249 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-249 - 2020/01/17 10:56:47.305429 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServerPeriodic-node-249 (Addr: tcp/127.0.0.1:16693) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-249 - 2020/01/17 10:56:47.305433 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServerPeriodic-node-249.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-252 - 2020/01/17 10:56:47.342810 [DEBUG] tlsutil: Update with version 1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-252 - 2020/01/17 10:56:47.343217 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
writer.go:21: 2020/01/17 10:56:47 [WARN] raft: no known peers, aborting election
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: Initial configuration (index=0): []
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.550209 [DEBUG] consul: Skipping self join check for "TestAutopilot_CleanupDeadServerPeriodic-node-235" since the cluster is too small
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: Node at 127.0.0.1:16702 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-252 - 2020/01/17 10:56:47.564213 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-252.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-252 - 2020/01/17 10:56:47.571916 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-252 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-252 - 2020/01/17 10:56:47.572090 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServerPeriodic-node-252 (Addr: tcp/127.0.0.1:16702) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-252 - 2020/01/17 10:56:47.572121 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServerPeriodic-node-252.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.573041 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:16652
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.573092 [DEBUG] memberlist: Stream connection from=127.0.0.1:64600
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.573567 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-242 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.573748 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServerPeriodic-node-242 (Addr: tcp/127.0.0.1:16672) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.573920 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-235 127.0.0.1
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: Updating configuration with AddNonvoter (74671791-9cd2-2504-27e4-f9571a9d380e, 127.0.0.1:16672) to [{Suffrage:Voter ID:b2142876-cec7-836d-168b-42b00fd9c321 Address:127.0.0.1:16651} {Suffrage:Nonvoter ID:74671791-9cd2-2504-27e4-f9571a9d380e Address:127.0.0.1:16672}]
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.574020 [INFO] consul: Adding LAN server TestAutopilot_CleanupDeadServerPeriodic-node-235 (Addr: tcp/127.0.0.1:16651) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.575127 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:16674
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:47.575207 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:16652
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.575258 [DEBUG] memberlist: Stream connection from=127.0.0.1:64601
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.575151 [ERR] memberlist: Error accepting TCP connection: accept tcp 127.0.0.1:16652: accept: too many open files
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.575315 [DEBUG] memberlist: Stream connection from=127.0.0.1:64605
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:47.575400 [DEBUG] memberlist: Failed to join 127.0.0.1: EOF
helper_test.go:128: 1 error occurred:
* Failed to join 127.0.0.1: EOF


log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-252 - 2020/01/17 10:56:47.575434 [INFO] consul: shutting down server
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-252 - 2020/01/17 10:56:47.575464 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.575699 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-235.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.575733 [INFO] serf: EventMemberJoin: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.575784 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServerPeriodic-node-235.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.575875 [DEBUG] consul: Successfully performed flood-join for "TestAutopilot_CleanupDeadServerPeriodic-node-242" at 127.0.0.1:16674
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.575889 [INFO] consul: Handled member-join event for server "TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1" in area "wan"
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.576013 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:16653
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.576756 [DEBUG] consul: Successfully performed flood-join for "TestAutopilot_CleanupDeadServerPeriodic-node-235" at 127.0.0.1:16653
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-252 - 2020/01/17 10:56:47.591737 [WARN] serf: Shutdown without a Leave
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: Added peer 74671791-9cd2-2504-27e4-f9571a9d380e, starting replication
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.599237 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.599281 [INFO] consul: member 'TestAutopilot_CleanupDeadServerPeriodic-node-242' joined, marking health alive
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.600481 [DEBUG] raft-net: 127.0.0.1:16672 accepted connection from: 127.0.0.1:64607
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-252 - 2020/01/17 10:56:47.605969 [INFO] manager: shutting down
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.621362 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.621406 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.621385 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-235.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.624433 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-235": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.624474 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.627458 [DEBUG] raft-net: 127.0.0.1:16672 accepted connection from: 127.0.0.1:64609
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.630874 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.630928 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.632117 [DEBUG] serf: messageUserEventType: consul:new-leader
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.634159 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-235.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.634175 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.634185 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-235.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.634193 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.634199 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1
writer.go:21: 2020/01/17 10:56:47 [WARN] raft: Failed to get previous log: 13 log not found (last: 0)
writer.go:21: 2020/01/17 10:56:47 [WARN] raft: AppendEntries to {Nonvoter 74671791-9cd2-2504-27e4-f9571a9d380e 127.0.0.1:16672} rejected, sending older logs (next: 1)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.641574 [DEBUG] serf: messageUserEventType: consul:new-leader
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.641698 [DEBUG] serf: messageUserEventType: consul:new-leader
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.641719 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.641732 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.641839 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.710208 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.716934 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.726339 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.734026 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-235.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.734069 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.740661 [DEBUG] serf: messageUserEventType: consul:new-leader
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.740750 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-249 - 2020/01/17 10:56:47.754984 [INFO] consul: shutting down server
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-249 - 2020/01/17 10:56:47.755022 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.759472 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-249 - 2020/01/17 10:56:47.778904 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.809323 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.817899 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:47.825933 [DEBUG] serf: messageJoinType: TestAutopilot_CleanupDeadServerPeriodic-node-242
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-249 - 2020/01/17 10:56:47.851309 [INFO] manager: shutting down
writer.go:21: 2020/01/17 10:56:47 [INFO] raft: pipelining replication to peer {Nonvoter 74671791-9cd2-2504-27e4-f9571a9d380e 127.0.0.1:16672}
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.897526 [DEBUG] consul: Skipping self join check for "TestAutopilot_CleanupDeadServerPeriodic-node-235" since the cluster is too small
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.922906 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:47.959927 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:48.085977 [INFO] consul: shutting down server
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:48.086035 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:48.157265 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-246 - 2020/01/17 10:56:48.183471 [INFO] manager: shutting down
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.183806 [DEBUG] consul: Skipping self join check for "TestAutopilot_CleanupDeadServerPeriodic-node-235" since the cluster is too small
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.184290 [INFO] autopilot: Promoting Server (ID: "74671791-9cd2-2504-27e4-f9571a9d380e" Address: "127.0.0.1:16672") to voter
writer.go:21: 2020/01/17 10:56:48 [INFO] raft: Updating configuration with AddStaging (74671791-9cd2-2504-27e4-f9571a9d380e, 127.0.0.1:16672) to [{Suffrage:Voter ID:b2142876-cec7-836d-168b-42b00fd9c321 Address:127.0.0.1:16651} {Suffrage:Voter ID:74671791-9cd2-2504-27e4-f9571a9d380e Address:127.0.0.1:16672}]
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.209233 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.314843 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:48.421305 [INFO] consul: shutting down server
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:48.421377 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:48.449152 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.489926 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.490599 [DEBUG] memberlist: Failed ping: TestAutopilot_CleanupDeadServerPeriodic-node-242 (timeout reached)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:48.494454 [INFO] manager: shutting down
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:48.494496 [ERR] raft-net: Failed to decode incoming command: transport shutdown
writer.go:21: 2020/01/17 10:56:48 [INFO] raft: aborting pipeline replication to peer {Nonvoter 74671791-9cd2-2504-27e4-f9571a9d380e 127.0.0.1:16672}
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.515225 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-242 - 2020/01/17 10:56:48.517734 [ERR] raft-net: Failed to decode incoming command: transport shutdown
writer.go:21: 2020/01/17 10:56:48 [ERROR] raft: Failed to heartbeat to 127.0.0.1:16672: EOF
writer.go:21: 2020/01/17 10:56:48 [WARN] raft: could not get configuration for Stats: raft is already shutdown
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.540461 [INFO] memberlist: Suspect TestAutopilot_CleanupDeadServerPeriodic-node-242 has failed, no acks received
writer.go:21: 2020/01/17 10:56:48 [ERROR] raft: Failed to AppendEntries to {Nonvoter 74671791-9cd2-2504-27e4-f9571a9d380e 127.0.0.1:16672}: dial tcp ->127.0.0.1:16672: connect: connection refused
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.583643 [DEBUG] memberlist: Failed ping: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1 (timeout reached)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.584027 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": rpc error making call: EOF
writer.go:21: 2020/01/17 10:56:48 [WARN] raft: Failed to contact 74671791-9cd2-2504-27e4-f9571a9d380e in 100.652961ms
writer.go:21: 2020/01/17 10:56:48 [WARN] raft: Failed to contact quorum of nodes, stepping down
writer.go:21: 2020/01/17 10:56:48 [INFO] raft: Node at 127.0.0.1:16651 [Follower] entering Follower state (Leader: "")
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.595499 [ERR] consul: failed to wait for barrier: leadership lost while committing log
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.595518 [DEBUG] consul: shutting down leader loop
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.595556 [DEBUG] leader: stopping CA root pruning routine
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.595576 [DEBUG] leader: stopped CA root pruning routine
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.610165 [WARN] consul: error getting server health from "TestAutopilot_CleanupDeadServerPeriodic-node-242": context deadline exceeded
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.610339 [INFO] consul: cluster leadership lost
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.634666 [INFO] memberlist: Suspect TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1 has failed, no acks received
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.695052 [DEBUG] memberlist: Failed ping: TestAutopilot_CleanupDeadServerPeriodic-node-242 (timeout reached)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.741648 [INFO] memberlist: Marking TestAutopilot_CleanupDeadServerPeriodic-node-242 as failed, suspect timeout reached (0 peer confirmations)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.741790 [INFO] serf: EventMemberFailed: TestAutopilot_CleanupDeadServerPeriodic-node-242 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.741888 [INFO] consul: Removing LAN server TestAutopilot_CleanupDeadServerPeriodic-node-242 (Addr: tcp/127.0.0.1:16672) (DC: dc1)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.780584 [INFO] consul: shutting down server
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.780742 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.784915 [DEBUG] memberlist: Failed ping: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1 (timeout reached)
writer.go:21: 2020/01/17 10:56:48 [ERROR] raft: Failed to heartbeat to 127.0.0.1:16672: read tcp 127.0.0.1:64642->127.0.0.1:16672: read: connection reset by peer
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.821455 [WARN] serf: Shutdown without a Leave
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.835365 [INFO] memberlist: Marking TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1 as failed, suspect timeout reached (0 peer confirmations)
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.835406 [INFO] serf: EventMemberFailed: TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1 127.0.0.1
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.842457 [INFO] memberlist: Suspect TestAutopilot_CleanupDeadServerPeriodic-node-242 has failed, no acks received
writer.go:21: 2020/01/17 10:56:48 [WARN] raft: Heartbeat timeout from "" reached, starting election
writer.go:21: 2020/01/17 10:56:48 [INFO] raft: Node at 127.0.0.1:16651 [Candidate] entering Candidate state in term 3
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.913105 [INFO] manager: shutting down
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:48.938425 [INFO] memberlist: Suspect TestAutopilot_CleanupDeadServerPeriodic-node-242.dc1 has failed, no acks received
log.go:172: TestAutopilot_CleanupDeadServerPeriodic-node-235 - 2020/01/17 10:56:49.141792 [WARN] raft: Unable to get address for server id 74671791-9cd2-2504-27e4-f9571a9d380e, using fallback address 127.0.0.1:16672: Could not find address for server id 74671791-9cd2-2504-27e4-f9571a9d380e
writer.go:21: 2020/01/17 10:56:49 [ERROR] raft: Failed to make RequestVote RPC to {Voter 74671791-9cd2-2504-27e4-f9571a9d380e 127.0.0.1:16672}: dial tcp ->127.0.0.1:16672: connect: connection refused
FAIL agent/consul.TestAutopilot_CleanupDeadServerPeriodic (3.03s)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.