after changing the domain from default consul to custom cannot reach any service or node
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
After changing the domain from default to lon in the configurations of all the consul servers and client agents, I was expecting to resolve the service redis using `dig redis.service.lon `
here is the configuration I am using
```
{
"node_name": "rockpix",
"bind_addr": "{{GetInterfaceIP \"enp1s0\"}}",
"ui": false,
"client_addr": "127.0.0.1 {{ GetPrivateIPs }}",
"data_dir": "/opt/consul",
"datacenter": "lon",
"domain": "lon",
"disable_remote_exec": false,
"bootstrap_expect": 3,
"log_level": "INFO",
"enable_syslog": true,
"enable_debug": true,
"server": true,
"leave_on_terminate": false,
"skip_leave_on_interrupt": true,
"rejoin_after_leave": true,
"advertise_addr": "{{GetInterfaceIP \"enp1s0\"}}",
"retry_join": [
"192.168.1.11",
"192.168.1.40"
]
}
```
for other nodes the value in retry_join changes. that's it. the only field I added is "domain": "lon"
#### Reproduction Steps
Steps to reproduce this issue, eg:
1. Stop all the consul cluster nodes
1. Change the "domian": "lon" in all the consul server and client nodes.
1. restart consul servers and client nodes.
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = c976ffd2
version = 1.10.3
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 192.168.1.40:8300
server = true
raft:
applied_index = 31140
commit_index = 31140
fsm_pending = 0
last_contact = 4.879ms
last_log_index = 31140
last_log_term = 266
last_snapshot_index = 30642
last_snapshot_term = 266
latest_configuration = [{Suffrage:Voter ID:79321fb9-b766-8b1b-3915-0991c9f8750d Address:192.168.1.11:8300} {Suffrage:Voter ID:bced0c51-052d-41f4-cf63-3d209fc2163d Address:192.168.1.40:8300} {Suffrage:Voter ID:2b8c1d1f-daa3-3719-de1a-1cff592a673c Address:192.168.1.46:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 266
runtime:
arch = arm
cpu_count = 1
goroutines = 105
max_procs = 1
os = linux
version = go1.16.7
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 38
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 13389
members = 4
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5576
members = 3
query_queue = 0
query_time = 1
```
Server info
```# consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 5
build:
prerelease =
revision = c976ffd2
version = 1.10.3
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 192.168.1.40:8300
server = true
raft:
applied_index = 31135
commit_index = 31135
fsm_pending = 0
last_contact = 75.699285ms
last_log_index = 31135
last_log_term = 266
last_snapshot_index = 16387
last_snapshot_term = 261
latest_configuration = [{Suffrage:Voter ID:79321fb9-b766-8b1b-3915-0991c9f8750d Address:192.168.1.11:8300} {Suffrage:Voter ID:bced0c51-052d-41f4-cf63-3d209fc2163d Address:192.168.1.40:8300} {Suffrage:Voter ID:2b8c1d1f-daa3-3719-de1a-1cff592a673c Address:192.168.1.46:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 266
runtime:
arch = amd64
cpu_count = 4
goroutines = 116
max_procs = 4
os = linux
version = go1.16.7
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 38
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 13389
members = 4
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5576
members = 3
query_queue = 0
query_time = 1
```
### Operating system and Environment details
OS, Architecture, and any other information you can provide about the environment.
Client : Linux raspberrypi 5.10.63+ #1459 Wed Oct 6 16:40:27 BST 2021 armv6l GNU/Linux
Server1 : Linux rockpi 4.4.154-95-rockchip-gd2ab1f26e1b3 #1 SMP Mon Oct 21 06:42:35 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
Server2: Linux rockpix 5.11.0-38-generic #42~20.04.1-Ubuntu SMP Tue Sep 28 20:41:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Server3: Linux rockpro64 5.10.63-rockchip64 #21.08.2 SMP PREEMPT Wed Sep 8 10:57:23 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
### Log Fragments
```
Nov 08 19:52:19 raspberrypi consul[13563]: 2021-11-08T19:52:19.464Z [INFO] agent: Joining cluster...: cluster=LAN
Nov 08 19:52:19 raspberrypi consul[13563]: 2021-11-08T19:52:19.464Z [INFO] agent: (LAN) joining: lan_addresses=[192.168.1.11, 192.168.1.40]
Nov 08 19:52:19 raspberrypi consul[13563]: 2021-11-08T19:52:19.430Z [INFO] agent.server: Adding LAN server: server="rockpix (Addr: tcp/192.168.1.11:8300) (DC: lon)"
Nov 08 19:52:19 raspberrypi consul[13563]: 2021-11-08T19:52:19.483Z [INFO] agent.server.serf.wan: serf: Re-joined to previously known node: rockpro64.lon: 192.168.1.40:8302
Nov 08 19:52:19 raspberrypi consul[13563]: 2021-11-08T19:52:19.430Z [INFO] agent.server: New leader elected: payload=rockpro64
Nov 08 19:52:19 raspberrypi consul[13563]: 2021-11-08T19:52:19.363Z [INFO] agent.server: Handled event for server in area: event=member-join server=rockpro64.lon area=wan
Nov 08 19:52:19 raspberrypi consul[13563]: 2021-11-08T19:52:19.443Z [INFO] agent.server: Handled event for server in area: event=member-join server=rockpix.lon area=wan
Nov 08 19:52:19 raspberrypi consul[13563]: 2021-11-08T19:52:19.387Z [WARN] agent: grpc: addrConn.createTransport failed to connect to {lon-192.168.1.46:8300 0 pizero.lon }. Err :connection error: des
Nov 08 19:52:19 raspberrypi consul[13563]: 2021-11-08T19:52:19.387Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
```
Contributor guide
Research direction
Start by reproducing the failure with the provided Consul configuration and `dig redis.service.lon`, then compare the client and server logs around custom-domain membership and DNS resolution. Done means services and nodes are reachable after changing the domain from the default to `lon`.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100