Joining cluster with duplicate Node Id causes gossip failures

#### Overview of the Issue

Consul allows a node to partially join the cluster with a duplicate Node Id. This puts the cluster into a someone indeterminate state where the node catalog may be unstable, and consul agents report failures joining the cluster. This can happen when restoring a node from a snapshot while the original node is still alive, or via any mechanism that causes the Node Id to be replicated.

In brief the problem seems to be an abstraction issue. The joining of the cluster is handled at the Serf/Memberlist level, which has no concept of NodeId (it's just abstract metadata), but Consul requires and attempts to enforce the uniqueness of NodeIds, but doesn't have the ability to block the join until it's too late and the duplicate ID is injected into gossip.

This leads to inconsistency between the view at the Serf layer (everything is fully joined up and gossiping) and the Consul (duplicate Node Ids cause trouble with the catalog).

During this time the cluster is somewhat functional (non duplicated nodes seem to be part of the cluster, and their catalog entries, etc seem correct). The cluster seems to recover smoothly when one of the duplicates is removed.

This is abstracted from a customer issue: [consul-dup-node-id](https://go.hashi.co/consul-dup-node-id)

##### Details

* All agents complain of duplicate node ids, with messages like:
`Member 'dup-client' has conflicting node ID '23eb77d9-7114-4ed8-8502-fb9ba6ec7593' with this agent's ID` or
`Member 'dup-client' has conflicting node ID '23eb77d9-7114-4ed8-8502-fb9ba6ec7593' with member 'consul-client'`

New clients may report failure to join.

* The members view contains multiple nodes with the same id:

* The consul node catalog shows one of the duplicated nodes (but this may flip-flop over time)

#### Reproduction Steps
Using the `docker-compose.yml` file

```
version: '3.7'

services:

consul-server:
image: hashicorp/consul:1.12.0
container_name: consul-server
networks:
- consul
ports:
- "8500:8500"
command: "agent -server -bootstrap -node consul-server -client 0.0.0.0"

consul-client:
image: hashicorp/consul:1.12.0
container_name: consul-client
networks:
- consul
command: "agent -node-id=23eb77d9-7114-4ed8-8502-fb9ba6ec7593 -node consul-client -retry-join consul-server"

dup-client:
image: hashicorp/consul:1.12.0
container_name: dup-client
networks:
- consul
command: "agent -node-id=23eb77d9-7114-4ed8-8502-fb9ba6ec7593 -node dup-client -retry-join consul-server"

new-client:
image: hashicorp/consul:1.12.0
container_name: new-client
networks:
- consul
command: "agent -node new-client -retry-join consul-server"

networks:
consul:
driver: bridge

```
Run `docker-compose up -d consul-{server,client}` to bring up the first two nodes. It's helpful to run `docker-compose logs -f` in another window.

Run `docker-compose up -d dup-client` to bring up the conflicting client. Observe all the agents log duplicate errors.

Run `docker-compose up -d new-client` to bring up a new client. It will also log an error about duplicate nodes in the cluster.

The cluster is now in a strange state.
A `consul members -detailed` on the server will show all the nodes, (see above)
```
consul-server 172.27.0.3:8301 alive acls=0,ap=default,bootstrap=1,build=1.12.0:09a8cdb4,dc=dc1,ft_fs=1,ft_si=1,id=86ffd7f1-275b-5fa8-d11b-bb5a66d3a60b,port=8300,raft_vsn=3,role=consul,segment=,vsn=2,vsn_max=3,vsn_min=2,wan_join_port=8302
consul-client 172.27.0.2:8301 alive ap=default,build=1.12.0:09a8cdb4,dc=dc1,id=23eb77d9-7114-4ed8-8502-fb9ba6ec7593,role=node,segment=,vsn=2,vsn_max=3,vsn_min=2
dup-client 172.27.0.4:8301 alive ap=default,build=1.12.0:09a8cdb4,dc=dc1,id=23eb77d9-7114-4ed8-8502-fb9ba6ec7593,role=node,segment=,vsn=2,vsn_max=3,vsn_min=2
new-client 172.27.0.5:8301 alive ap=default,build=1.12.0:09a8cdb4,dc=dc1,id=f13aa89a-ad2c-ed0d-a0ec-6c9028cb5cef,role=node,segment=,vsn=2,vsn_max=3,vsn_min=2
```
Note the duplicate id in the meta field.

But consul-client and dup-client will omit the other's information.

A `consul catalog nodes -detailed` will list either the consul-client or the dup-client, but not both, and may shift from one to the other periodically.
```
Node ID Address DC TaggedAddresses Meta
consul-server 86ffd7f1-275b-5fa8-d11b-bb5a66d3a60b 172.27.0.3 dc1 lan=172.27.0.3, lan_ipv4=172.27.0.3, wan=172.27.0.3, wan_ipv4=172.27.0.3 consul-network-segment=
dup-client 23eb77d9-7114-4ed8-8502-fb9ba6ec7593 172.27.0.4 dc1
new-client f13aa89a-ad2c-ed0d-a0ec-6c9028cb5cef 172.27.0.5 dc1 lan=172.27.0.5, lan_ipv4=172.27.0.5, wan=172.27.0.5, wan_ipv4=172.27.0.5 consul-network-segment=
```

### Consul info for both Client and Server
This can be reproduced in Consul 1.9.x and 1.12.0

### Operating system and Environment details

The repro here was done in docker, but the example came from a live environment.

### Log Fragments
```
consul-server | 2022-05-11T02:39:01.855Z [ERROR] agent.server.memberlist.lan: memberlist: Failed push/pull merge: Member 'dup-client' has conflicting node ID '23eb77d9-7114-4ed8-8502-fb9ba6ec7593' with member 'consul-client' from=172.27.0.5:37878
new-client | 2022-05-11T02:39:01.855Z [WARN] agent: (LAN) couldn't join: number_of_nodes=0 error="1 error occurred:
new-client | * Failed to join 172.27.0.3:8301: Member 'consul-client' has conflicting node ID '23eb77d9-7114-4ed8-8502-fb9ba6ec7593' with member 'dup-client'"
new-client | 2022-05-11T02:39:01.855Z [WARN] agent: Join cluster failed, will retry: cluster=LAN retry_interval=30s error=
consul-client | 2022-05-11T02:39:10.196Z [WARN] agent.client.memberlist.lan: memberlist: ignoring alive message for 'dup-client': Member 'dup-client' has conflicting node ID '23eb77d9-7114-4ed8-8502-fb9ba6ec7593' with this agent's ID
dup-client | 2022-05-11T02:39:10.751Z [INFO] agent: Synced node info dup-client | 2022-05-11T02:39:13.823Z [INFO] agent: (LAN) joining: lan_
addresses=[consul-server]
dup-client | 2022-05-11T02:39:13.824Z [WARN] agent: (LAN) couldn't join: number_of_nodes=0 error="1 error occurred:
dup-client | * Failed to join 172.27.0.3:8301: Member 'consul-client' has conflicting node ID '23eb77d9-7114-4ed8-8502-fb9ba6ec7593' with this ag
ent's ID
```

Start by running the duplicate-node reproduction from docker-compose.yml and examining the Consul agent's LAN join path through Serf/Memberlist. Compare the members and catalog outputs before and after the duplicate joins. Done means duplicate Node IDs no longer destabilize the catalog or prevent other clients from joining.

Written by the indexing model from the issue text.