Unstable network issue

Hi there,

I'm using Serf (version **0.8.1**) as a library to create a clustered application that can be deployed via containers inside Docker Swarm clusters.

Typical setup is:

* The creation of a Docker network using the overlay driver
* Deployment of that application as a global Swarm service inside the previously created overlay network, which will create a container on each node of the Swarm cluster.

This setup is working well on some infrastructures and providers but is having communication issues inside an AWS VPC.

Now, I've looked at lot of issues regarding network instability in AWS when using Consul (which is using Serf I believe), but no solution to the problem.

All the issues seem to have been closed since the introduction of the Lifeguard features, but I believe that the problem is still here.

I'll try to give as many info as I can.

## Deployment

After deploying the agent as a global service inside a 3 node cluster ([using this stack file](https://portainer.io/download/portainer-agent-stack.yml)), we can see that each container start without problem and can discover each other:

Container 1:
```
...@ip-172-37-13-17... | 2018/05/18 12:16:34 [INFO] serf: EventMemberJoin: 7ca75632a5c2 10.0.0.6
...@ip-172-37-13-17... | 2018/05/18 12:16:34 [INFO] serf: EventMemberJoin: b7b8fca7ba23 10.0.0.7
...@ip-172-37-13-17... | 2018/05/18 12:16:34 [INFO] - Starting Portainer agent version 1.0.0-dev on 0.0.0.0:9001
...@ip-172-37-13-17... | 2018/05/18 12:16:54 [INFO] serf: EventMemberJoin: b6b20d6971cc 10.0.0.8
```

Container 2:
```
...@ip-172-37-48-133.. | 2018/05/18 12:16:33 [INFO] serf: EventMemberJoin: b7b8fca7ba23 10.0.0.7
...@ip-172-37-48-133.. | 2018/05/18 12:16:34 [INFO] - Starting Portainer agent version 1.0.0-dev on 0.0.0.0:9001
...@ip-172-37-48-133.. | 2018/05/18 12:16:34 [INFO] serf: EventMemberJoin: 7ca75632a5c2 10.0.0.6
...@ip-172-37-48-133.. | 2018/05/18 12:16:54 [INFO] serf: EventMemberJoin: b6b20d6971cc 10.0.0.8
```

Container 3:
```
...@ip-172-37-17-77.. | 2018/05/18 12:16:53 [INFO] serf: EventMemberJoin: b6b20d6971cc 10.0.0.8
...@ip-172-37-17-77.. | 2018/05/18 12:16:54 [INFO] serf: EventMemberJoin: 7ca75632a5c2 10.0.0.6
...@ip-172-37-17-77.. | 2018/05/18 12:16:54 [INFO] serf: EventMemberJoin: b7b8fca7ba23 10.0.0.7
...@ip-172-37-17-77.. | 2018/05/18 12:16:54 [INFO] - Starting Portainer agent version 1.0.0-dev on 0.0.0.0:9001

```

## Troubles

Now, **without any activity nor any interaction with any of the application**, the communication between each container starts to fail after a few minutes:

Container 1:
```
...@ip-172-37-13-17... | 2018/05/18 12:20:14 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.6:43834->10.0.0.8:7946: i/o timeout
...@ip-172-37-13-17... | 2018/05/18 12:20:14 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-13-17... | 2018/05/18 12:20:29 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.6:43838->10.0.0.8:7946: i/o timeout
...@ip-172-37-13-17... | 2018/05/18 12:20:29 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-13-17... | 2018/05/18 12:20:39 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.6:43848->10.0.0.8:7946: i/o timeout
...@ip-172-37-13-17... | 2018/05/18 12:20:39 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-13-17... | 2018/05/18 12:20:44 [INFO] memberlist: Marking b6b20d6971cc as failed, suspect timeout reached (0 peer confirmations)
...@ip-172-37-13-17... | 2018/05/18 12:20:44 [INFO] serf: EventMemberFailed: b6b20d6971cc 10.0.0.8
...@ip-172-37-13-17... | 2018/05/18 12:21:04 [INFO] serf: attempting reconnect to b6b20d6971cc 10.0.0.8:7946
...@ip-172-37-13-17... | 2018/05/18 12:22:04 [INFO] serf: attempting reconnect to b6b20d6971cc 10.0.0.8:7946
...@ip-172-37-13-17... | 2018/05/18 12:23:04 [INFO] serf: attempting reconnect to b6b20d6971cc 10.0.0.8:7946
...@ip-172-37-13-17... | 2018/05/18 12:25:29 [WARN] memberlist: Refuting a suspect message (from: b7b8fca7ba23)
...@ip-172-37-13-17... | 2018/05/18 12:25:29 [INFO] serf: EventMemberJoin: b6b20d6971cc 10.0.0.8
...@ip-172-37-13-17... | 2018/05/18 12:26:04 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.6:44014->10.0.0.8:7946: i/o timeout
...@ip-172-37-13-17... | 2018/05/18 12:26:04 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-13-17... | 2018/05/18 12:28:54 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.6:44100->10.0.0.8:7946: i/o timeout
...@ip-172-37-13-17... | 2018/05/18 12:28:54 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-13-17... | 2018/05/18 12:30:24 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.6:44154->10.0.0.8:7946: i/o timeout
...@ip-172-37-13-17... | 2018/05/18 12:30:24 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-13-17... | 2018/05/18 12:30:34 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.6:44158->10.0.0.8:7946: i/o timeout
...@ip-172-37-13-17... | 2018/05/18 12:30:34 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-13-17... | 2018/05/18 12:30:44 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.6:44170->10.0.0.8:7946: i/o timeout
...@ip-172-37-13-17... | 2018/05/18 12:30:44 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-13-17... | 2018/05/18 12:30:54 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.6:44172->10.0.0.8:7946: i/o timeout
...@ip-172-37-13-17... | 2018/05/18 12:30:54 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-13-17... | 2018/05/18 12:30:54 [INFO] memberlist: Marking b6b20d6971cc as failed, suspect timeout reached (0 peer confirmations)
...@ip-172-37-13-17... | 2018/05/18 12:30:54 [INFO] serf: EventMemberFailed: b6b20d6971cc 10.0.0.8
...@ip-172-37-13-17... | 2018/05/18 12:31:04 [INFO] serf: attempting reconnect to b6b20d6971cc 10.0.0.8:7946
```

Container 2:
```
...@ip-172-37-48-133.. | 2018/05/18 12:20:28 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.7:46038->10.0.0.8:7946: i/o timeout
...@ip-172-37-48-133.. | 2018/05/18 12:20:28 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-48-133.. | 2018/05/18 12:20:38 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.7:46044->10.0.0.8:7946: i/o timeout
...@ip-172-37-48-133.. | 2018/05/18 12:20:38 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-48-133.. | 2018/05/18 12:20:44 [INFO] memberlist: Marking b6b20d6971cc as failed, suspect timeout reached (0 peer confirmations)
...@ip-172-37-48-133.. | 2018/05/18 12:20:44 [INFO] serf: EventMemberFailed: b6b20d6971cc 10.0.0.8
...@ip-172-37-48-133.. | 2018/05/18 12:21:04 [INFO] serf: attempting reconnect to b6b20d6971cc 10.0.0.8:7946
...@ip-172-37-48-133.. | 2018/05/18 12:22:04 [INFO] serf: attempting reconnect to b6b20d6971cc 10.0.0.8:7946
...@ip-172-37-48-133.. | 2018/05/18 12:23:04 [INFO] serf: attempting reconnect to b6b20d6971cc 10.0.0.8:7946
...@ip-172-37-48-133.. | 2018/05/18 12:24:04 [INFO] serf: attempting reconnect to b6b20d6971cc 10.0.0.8:7946
...@ip-172-37-48-133.. | 2018/05/18 12:25:04 [INFO] serf: attempting reconnect to b6b20d6971cc 10.0.0.8:7946
...@ip-172-37-48-133.. | 2018/05/18 12:25:29 [INFO] serf: EventMemberJoin: b6b20d6971cc 10.0.0.8
...@ip-172-37-48-133.. | 2018/05/18 12:25:53 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.7:46336->10.0.0.8:7946: i/o timeout
...@ip-172-37-48-133.. | 2018/05/18 12:25:53 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-48-133.. | 2018/05/18 12:26:03 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.7:46348->10.0.0.8:7946: i/o timeout
...@ip-172-37-48-133.. | 2018/05/18 12:26:03 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-48-133.. | 2018/05/18 12:26:12 [WARN] memberlist: Refuting a suspect message (from: b6b20d6971cc)
...@ip-172-37-48-133.. | 2018/05/18 12:28:55 [WARN] memberlist: Refuting a suspect message (from: b6b20d6971cc)
...@ip-172-37-48-133.. | 2018/05/18 12:30:28 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.7:46584->10.0.0.8:7946: i/o timeout
...@ip-172-37-48-133.. | 2018/05/18 12:30:28 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-48-133.. | 2018/05/18 12:30:38 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.7:46596->10.0.0.8:7946: i/o timeout
...@ip-172-37-48-133.. | 2018/05/18 12:30:38 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-48-133.. | 2018/05/18 12:30:48 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.7:46604->10.0.0.8:7946: i/o timeout
...@ip-172-37-48-133.. | 2018/05/18 12:30:48 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-48-133.. | 2018/05/18 12:30:54 [INFO] memberlist: Marking b6b20d6971cc as failed, suspect timeout reached (0 peer confirmations)
...@ip-172-37-48-133.. | 2018/05/18 12:30:54 [INFO] serf: EventMemberFailed: b6b20d6971cc 10.0.0.8
...@ip-172-37-48-133.. | 2018/05/18 12:30:58 [ERR] memberlist: Failed fallback ping: read tcp 10.0.0.7:46608->10.0.0.8:7946: i/o timeout
...@ip-172-37-48-133.. | 2018/05/18 12:30:58 [INFO] memberlist: Suspect b6b20d6971cc has failed, no acks received
...@ip-172-37-48-133.. | 2018/05/18 12:30:59 [INFO] serf: attempting reconnect to b6b20d6971cc 10.0.0.8:7946
```

Container 3:
```
...@ip-172-37-17-77.. | 2018/05/18 12:23:40 [ERR] memberlist: Push/Pull with 7ca75632a5c2 failed: dial tcp 10.0.0.6:7946: i/o timeout
...@ip-172-37-17-77.. | 2018/05/18 12:24:49 [ERR] memberlist: Failed to send ack: write tcp 10.0.0.8:7946->10.0.0.7:46038: i/o timeout from=10.0.0.7:46038
...@ip-172-37-17-77.. | 2018/05/18 12:24:59 [INFO] memberlist: Suspect 7ca75632a5c2 has failed, no acks received
...@ip-172-37-17-77.. | 2018/05/18 12:25:03 [ERR] memberlist: Failed to send ack: write tcp 10.0.0.8:7946->10.0.0.6:43834: i/o timeout from=10.0.0.6:43834
...@ip-172-37-17-77.. | 2018/05/18 12:25:28 [ERR] memberlist: Failed to send ack: write tcp 10.0.0.8:7946->10.0.0.6:43848: i/o timeout from=10.0.0.6:43848
...@ip-172-37-17-77.. | 2018/05/18 12:25:29 [ERR] memberlist: Failed to send ack: write tcp 10.0.0.8:7946->10.0.0.6:43838: i/o timeout from=10.0.0.6:43838
...@ip-172-37-17-77.. | 2018/05/18 12:25:29 [ERR] memberlist: Failed to send ack: write tcp 10.0.0.8:7946->10.0.0.7:46044: i/o timeout from=10.0.0.7:46044
...@ip-172-37-17-77.. | 2018/05/18 12:25:29 [WARN] memberlist: Refuting a suspect message (from: b6b20d6971cc)
...@ip-172-37-17-77.. | 2018/05/18 12:26:11 [WARN] memberlist: Refuting a suspect message (from: b7b8fca7ba23)
...@ip-172-37-17-77.. | 2018/05/18 12:26:11 [INFO] memberlist: Suspect b7b8fca7ba23 has failed, no acks received
...@ip-172-37-17-77.. | 2018/05/18 12:28:55 [INFO] memberlist: Suspect b7b8fca7ba23 has failed, no acks received
...@ip-172-37-17-77.. | 2018/05/18 12:28:55 [WARN] memberlist: Refuting a suspect message (from: 7ca75632a5c2)
...@ip-172-37-17-77.. | 2018/05/18 12:32:30 [ERR] memberlist: Push/Pull with b7b8fca7ba23 failed: dial tcp 10.0.0.7:7946: i/o timeout
```

I only pasted logs from a 10-min duration but the logs of each app is filled with similar errors.

## Usage

Here is how I used the Serf library inside the application (application built using go **1.9.5**):

```go
filter := &logutils.LevelFilter{
Levels: []logutils.LogLevel{"DEBUG", "INFO", "WARN", "ERROR"},
MinLevel: logutils.LogLevel("INFO"),
Writer: os.Stderr,
}

conf := serf.DefaultConfig()
conf.Init()
conf.MemberlistConfig.LogOutput = filter
conf.LogOutput = filter
conf.MemberlistConfig.AdvertiseAddr = advertiseAddr

cluster, err := serf.Create(conf)
if err != nil {
return err
}

_, err = cluster.Join([]string{joinAddr}, true)
if err != nil {
return err
}
```

I also tried to update the underlying memberlist configuration using `memberlist.DefaultWANConfig()`:

```go
conf := serf.DefaultConfig()
conf.Init()
conf.MemberlistConfig = memberlist.DefaultWANConfig()
conf.MemberlistConfig.LogOutput = filter
conf.LogOutput = filter
conf.MemberlistConfig.AdvertiseAddr = advertiseAddr

cluster, err := serf.Create(conf)
if err != nil {
return err
}

_, err = cluster.Join([]string{joinAddr}, true)
if err != nil {
return err
}
```

Same issue.

I'm keen to help you diagnose this issue by giving you any extra information you need.

Hope that this can be solved.