hashicorp / hashicorp/consul

Wan federation doesn't work with FQDN node name

Open
#12,614 1 comment 0 reactions 0 assignees View on GitHub
needs-discussion theme/federation-usability type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Related Topic on Hashicorp Discuss

[Connection failure in federation between VMs (primary) and kubernetes](https://discuss.hashicorp.com/t/connection-failure-in-federation-between-vms-primary-and-kubernetes/37201)

#### Overview of the Issue

I wanted to federate a consul cluster running in k3s (named: `dc2`) to a consul cluster running on VMs (named: `dc1`).
The nodes of the VM cluster have a node name in FQDN format (eg `ceph-1.hirsingue.infra.mydomain.fr`) which causes the following error in the logs (`dc2` consul server) and prevents the federation from continuing:

```
2022-03-22T08:42:47.170Z [INFO] agent: (WAN) joined: number_of_nodes=1
2022-03-22T08:42:47.170Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=WAN num_agents=1
2022-03-22T08:42:47.170Z [INFO] agent.server: Handled event for server in area: event=member-join server=ceph-2.hirsingue.infra.mydomain.fr.dc1 area=wan
2022-03-22T08:42:47.657Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 192.168.11.11:8302: node name does not encode a datacenter: ceph-2.hirsingue.infra.mydomain.fr.dc1
```

#### Reproduction Steps

This should be similar for any cluster type, but in my case :
- have a cluster in VMs up and running with node name in FQDN format.
eg: `ceph-1.hirsingue.infra.mydomain.fr`
- create and distribute TLS certificates on VM servers
eg: `consul tls cert create -server -dc dc1 -node ceph-1.hirsingue.infra.mydomain.fr`
- following this documentation: [Federation between VMs (primary) and Kubernetes](https://www.consul.io/docs/k8s/installation/multi-cluster/vms-and-kubernetes#kubernetes-as-the-secondary) to federate a mononode k3s.

### Consul info for dc1 and dc2 server

DC2 Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 37c7d06b
version = 1.11.2
consul:
acl = enabled
bootstrap = true
known_datacenters = 2
leader = true
leader_addr = 10.42.0.89:8300
server = true
raft:
applied_index = 18196
commit_index = 18196
fsm_pending = 0
last_contact = 0
last_log_index = 18196
last_log_term = 29
last_snapshot_index = 16385
last_snapshot_term = 29
latest_configuration = [{Suffrage:Voter ID:c6226cd1-b686-5e17-cf23-22bbc5d42e06 Address:10.42.0.89:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 29
runtime:
arch = amd64
cpu_count = 8
goroutines = 185
max_procs = 8
os = linux
version = go1.17.5
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 29
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 85
members = 2
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2533
members = 4
query_queue = 0
query_time = 1
```

DC1 Server info

```
root@ceph-1:~ # consul info
agent:
check_monitors = 0
check_ttls = 1
checks = 2
services = 2
build:
prerelease =
revision = 37c7d06b
version = 1.11.2
consul:
acl = enabled
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = 192.168.11.12:8300
server = true
raft:
applied_index = 445310
commit_index = 445310
fsm_pending = 0
last_contact = 13.990512ms
last_log_index = 445310
last_log_term = 7102
last_snapshot_index = 442448
last_snapshot_term = 7102
latest_configuration = [{Suffrage:Voter ID:d893d4cf-d43f-13e2-38b4-ee593d86d829 Address:192.168.11.11:8300} {Suffrage:Voter ID:0781d8ae-72a5-2a4b-9892-2aba24ea38f7 Address:192.168.11.12:8300} {Suffrage:Voter ID:23de9416-f021-b420-a9ef-6dd73313c54b Address:192.168.11.10:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 7102
runtime:
arch = amd64
cpu_count = 8
goroutines = 177
max_procs = 8
os = linux
version = go1.17.5
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 902
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 14504
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2533
members = 4
query_queue = 0
query_time = 1
```

### Operating system and Environment details

VMs OS:
```
Description: Ubuntu 21.04
Release: 21.04
```

Thanks for your help!

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by tracing the WAN memberlist handling associated with the reported “node name does not encode a datacenter” error, then reproduce federation with an FQDN node name; done means the clusters federate successfully without the logged failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, linux
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.