[HA] Show more info about the cluster status in the UI if it is failing

What did you do?

I configured 2 alertmanagers to talk to each other with only TCP port 9094 open between each other, UDP port 9094 blocked.

What did you expect to see?

An indication of an error state.

What did you see instead? Under which circumstances?

• Cluster status in the UI: "ready" (<-- in green)
• a reasonably clean daemon output (without --log.level=debug)

Environment

• System information:

  Linux 4.9.0-6-amd64 x86_64

• Alertmanager version:

  alertmanager, version 0.15.2 (branch: HEAD, revision: d19fae3bae451940b8470abb680cfdd59bfa7cfa)
  build user: root@3101e5b68a55
  build date: 20180814-10:53:39
  go version: go1.10.3

• Prometheus version: [N/A]

• Alertmanager configuration file:

Port 9094 was used as port in the cluster.listen-address and cluster.peer options.

• Prometheus configuration file:

insert configuration here (if relevant to the issue)

• Logs:

[WARN] memberlist: Was able to connect to 01CRQDCBH4NS9VTT139B9ECNFN but other probes failed, network may be misconfigured

Important: This warn message did not show until I ran both ends with --log.level=debug.

Start with the Alertmanager UI's cluster status and the HA configuration using cluster.listen-address and cluster.peer. Reproduce the setup with TCP port 9094 open and UDP port 9094 blocked, then compare the green "ready" status with the memberlist warning, including behavior at normal and debug log levels. Done means the UI indicates the failing cluster state.