Consul clusters in different versions are unable to resolve service tags cross DC
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
We have 4 consul clusters running version 1.2.1 .
We upgraded the first cluster (servers first, then clients) to version 1.7.8 and once the upgrade was completed we noticed that resolving of service tags from clients in the 1.7.8 cluster cross DC to the 1.2.1 version cluster return NXDOMAIN.
Looking at the logs (attached below), we see that for a plain service query there are 2 or more log lines (A, AAAA, ?MX)
but for tagged services we only see one log line (A)
We had to revert and downgrade the cluster back to version 1.2.1 to solve this
Is this intended?
What will be the procedure to upgrade assuming we have cross DC tagged services queries and we can only upgrade one datacenter at a time (we cannot upgrade all datacenters at the same day)?
#### Reproduction Steps
Steps to reproduce this issue, eg:
1. Create a cluster with n client nodes and 3 server nodes running version 1.2.1 (dc: vag1)
2. Create a cluster with n client nodes and 3 server nodes running version 1.7.8 (dc: vag2)
3. From one of the clients running version 1.7.8 run: `host ..service.vag1.consul`
### Consul info for 1.2.1 version Server
Client info
```
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 2
build:
prerelease =
revision = 39f93f01
version = 1.2.1
consul:
bootstrap = false
known_datacenters = 4
leader = true
leader_addr = :8300
server = true
raft:
applied_index = 1829650762
commit_index = 1829650762
fsm_pending = 0
last_contact = 0
last_log_index = 1829650762
last_log_term = 359458
last_snapshot_index = 1829649204
last_snapshot_term = 359458
latest_configuration = [{Suffrage:Voter ID:dde09378-4944-7018-9711-6be17deb00f3 Address::8300} {Suffrage:Voter ID:43b09dd3-b385-3b71-b4c5-9ef03df42ac6 Address::8300} {Suffrage:Voter ID:cfdd42a9-44e5-590a-1a50-e5e79935b30a Address::8300}]
latest_configuration_index = 1669000507
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 359458
runtime:
arch = amd64
cpu_count = 96
goroutines = 65997
max_procs = 96
os = linux
version = go1.10.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 154693
failed = 0
health_score = 0
intent_queue = 0
left = 2
member_time = 265197
members = 2897
query_queue = 0
query_time = 1299
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2093
members = 10
query_queue = 0
query_time = 1
```
### Consul info for 1.7.8 version Server
Server info
```
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 2
build:
prerelease =
revision = 9a5a1218
version = 1.7.8
consul:
acl = disabled
bootstrap = true
known_datacenters = 4
leader = true
leader_addr = :8300
server = true
raft:
applied_index = 11957906
commit_index = 11957906
fsm_pending = 0
last_contact = 0
last_log_index = 11957906
last_log_term = 8
last_snapshot_index = 11951901
last_snapshot_term = 6
latest_configuration =latest_configuration = [{Suffrage:Voter ID:f0800617-4070-9d22-3b80-48ea669e5fa5 Address::8300} {Suffrage:Voter ID:c56d717f-00ff-5f3d-a764-4459fc860f4f Address::8300} {Suffrage:Voter ID:b5320274-520b-08e2-7117-6d40bf0ac936 Address::8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 8
runtime:
arch = amd64
cpu_count = 2
goroutines = 652
max_procs = 2
os = linux
version = go1.13.15
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 30
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 713
members = 2650
query_queue = 0
query_time = 1322
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2093
members = 10
query_queue = 0
query_time = 1
```
### Operating system and Environment details
All consul servers run Ubuntu16, client are all ubuntu distro, various versions.
All DNS queries in our environment go through our internal DNS servers. This is also true for all internal consul DNS queries. The DNS slaves are configured to forward all `.consul` queries to the local agent at port 8600.
### Log Fragments
consul monitor logs from the dns servers:
```shell-session
(root@consul-client1-178.vag2~)$ host service1.service.vag1.consul
Using domain server:
Name: DNS1
Address: 172.16.38.12#53
Aliases:
service1.service.vag1.consul has address 172.16.38.15
```
```log
021-01-13T11:15:56.579-0500 [DEBUG] agent.dns: request served from client: name=service1.service.vag1.consul. type=A class=IN latency=157.290856ms client=127.0.0.1:46874 client_network=udp
2021-01-13T11:15:56.736-0500 [DEBUG] agent.dns: request served from client: name=service1.service.vag1.consul. type=AAAA class=IN latency=156.217508ms client=127.0.0.1:42954 client_network=udp
2021-01-13T11:15:56.894-0500 [DEBUG] agent.dns: request served from client: name=service1.service.vag1.consul. type=MX class=IN latency=156.605671ms client=127.0.0.1:56860 client_network=udp
```
```shell-session
(root@consul-client1-178:~)$ host test.service1.service.vag1.consul
Using domain server:
Name: DNS1
Address: 172.16.38.12#53
Aliases:
Host test.service1.service.vag1.consul not found: 3(NXDOMAIN)
```
```log
2021-01-13T11:15:58.482-0500 [DEBUG] agent.dns: request served from client: name=test.service1.service.vag1.consul. type=A class=IN latency=137.688429ms client=127.0.0.1:63095 client_network=udp
```
Contributor guide
Assessment
This issue has not been assessed yet.