hashicorp / hashicorp/consul

Consul/Vault registration results in DNS issues for vault.service.consul

Open
#21,953 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

I have a three-node cluster consisting of Nomad, Consul, and Vault.
I also use DNS forwarding of the .consul domain on my internal DNS servers.

While chasing down a DNS forwarder issue, specifically with vault.service.consul, I discovered that DNS answers are CNAMEs, and I do not get all the results of my unsealed Vault servers.

Queries against nomad.service.consul and consul.service.consul works fine.
I stood up a test BIND server during my troubleshooting, and I will show an example query below.

I believe this is a bug:
1. A query against vault.service.consul should output all unsealed healthy instances as documented [here](https://developer.hashicorp.com/vault/docs/configuration/service-registration/consul#configuration)
2. I believe the results should be A and not CNAME records. [Documentation](https://developer.hashicorp.com/consul/docs/services/discovery/dns-forwarding#workflow) that I could find shows the same.

Let me know if you require any further information.

```
$ named-checkconf -p
options {
directory "/var/cache/bind";
listen-on-v6 {
"any";
};
dnssec-validation no;
forwarders {
1.1.1.1;
9.9.9.9;
};
};
zone "consul" IN {
type forward;
forward only;
forwarders {
192.168.100.10 port 8600;
};
};
zone "." {
type hint;
file "/usr/share/dns/root.hints";
};
zone "localhost" {
type master;
file "/etc/bind/db.local";
};
zone "127.in-addr.arpa" {
type master;
file "/etc/bind/db.127";
};
zone "0.in-addr.arpa" {
type master;
file "/etc/bind/db.0";
};
zone "255.in-addr.arpa" {
type master;
file "/etc/bind/db.255";
};

$ dig @192.168.100.10 -p 8600 vault.service.consul

; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> @192.168.100.10 -p 8600 vault.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29059
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;vault.service.consul. IN A

;; ANSWER SECTION:
vault.service.consul. 0 IN CNAME prod-core-services03.

;; Query time: 1 msec
;; SERVER: 192.168.100.10#8600(192.168.100.10) (UDP)
;; WHEN: Mon Nov 18 21:40:59 UTC 2024
;; MSG SIZE rcvd: 83

$ dig @127.0.0.1 vault.service.consul

; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> @127.0.0.1 vault.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 7892
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 59186158b326193101000000673bb48caa5ecc2764a9541a (good)
;; QUESTION SECTION:
;vault.service.consul. IN A

;; ANSWER SECTION:
vault.service.consul. 0 IN CNAME prod-core-services03.

;; AUTHORITY SECTION:
. 2654 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2024111801 1800 900 604800 86400

;; Query time: 10 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Mon Nov 18 21:41:32 UTC 2024
;; MSG SIZE rcvd: 186
```

---

#### Reproduction Steps

1. Create a cluster with three server nodes
4. Unseal all Vault instances
5. Configure DNS forwarding
6. Perform a DNS query against vault.fqdn yields an NXDOMAIN response
7. Perform a dig against vault.service.consul yields one CNAME out of the total of three unsealed Vault instances.

### Consul info for both Client and Server

Client info

```
Output from client 'consul info' command here
```

```
Client agent HCL config
```

Server info

```
$ consul info
agent:
check_monitors = 0
check_ttls = 1
checks = 5
services = 5
build:
prerelease =
revision = cddc6181
version = 1.20.0
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 192.168.100.10:8300
server = true
raft:
applied_index = 1721463
commit_index = 1721463
fsm_pending = 0
last_contact = 0
last_log_index = 1721463
last_log_term = 138465
last_snapshot_index = 1714448
last_snapshot_term = 138463
latest_configuration = [{Suffrage:Voter ID:REDACTED Address:192.168.100.12:8300} {Suffrage:Voter ID:REDACTED Address:192.168.100.11:8300} {Suffrage:Voter ID:REDACTED Address:192.168.100.10:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 138465
runtime:
arch = arm64
cpu_count = 4
goroutines = 235
max_procs = 4
os = linux
version = go1.22.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 115521
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 46268
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2018
members = 3
query_queue = 0
query_time = 1
```

```
datacenter = "homelab"
data_dir = "/opt/consul/data"
encrypt = "REDACTED"
retry_join = [
"192.168.100.11",
"192.168.100.12"
]
server = true
bind_addr = "192.168.100.10"
client_addr = "127.0.0.1 192.168.100.10"
ui_config {
enabled = true
}
log_level = "INFO"
```

### Operating system and Environment details

Ubuntu 24.10
Vault 1.18
Consul 1.20

*Unsealed Vault instances*
```
$ consul catalog nodes -service=vault
Node ID Address DC
prod-core-services01 fdaa9e18 192.168.100.10 homelab
prod-core-services02 8de9943e 192.168.100.11 homelab
prod-core-services03 36374725 192.168.100.12 homelab
```

### Log Fragments

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported queries against Consul DNS on port 8600 and through the BIND forwarding configuration, using the three-node cluster and the unsealed Vault registrations described here. Done means determining why vault.service.consul returns one CNAME rather than all three healthy instances and confirming the expected DNS response through the forwarder.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.