hashicorp / hashicorp/consul

Reverse DNS fixed TTL to 0 (hard-coded), or always returning SERVFAIL

Open
#10,577 0 comments 1 reaction 0 assignees View on GitHub
theme/dns type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

Hi everyone,

I would like to share some reverse DNS behaviours that we faced, and to open discussions about some enhancement on consul reverse dns behaviour.

I put 2 issues in one ticket, but it is releated to the same code part

####The set-up

1. Few Powerdns recursors forwarding direct resolution and reverse dns to consul agent following [https://learn.hashicorp.com/tutorials/consul/dns-forwarding](https://learn.hashicorp.com/tutorials/consul/dns-forwarding)
2. About 1000 nodes on DC, 500 services...
3. No upstream recursors [https://www.consul.io/docs/agent/options#recursors](https://www.consul.io/docs/agent/options#recursors), meaning I consider Consul as authoritative on both his domain, and the reverse-dns ips allocated...

With nodes, and services growing, we start to see some higher CPU + Network usage coming from Consul agent on DNS server.

#### Overview of the (Issue 1)

After analysis, we observed :

Reverse-dns answers are always with 0 TTL. (hard-coded)

So even if we could play with DNS cache (some TTLs are changed for nodes, services in consul agent config...) it cannot be the case on reverse dns.

#### Reproduction Steps (issue 1)

Steps to reproduce this issue, eg:

1. Create a cluster with n client nodes n and n server nodes, and have an external dns to forward reverse dns to consul
1. Run `dig -x Ip address`
1. TTL is to 0

#### suggested behaviour:

Would it be possible to return Node TTL (in case a Node is found) or Service TTL in case a service is returned ? to be able to use caching mecanisms to reduce load on consul agent ?
If neither of these, could it be default minTTL value declared in SOA? or a specific added parameter?

#### Overview of the (Issue 2)

After analysis, we observed :

If you ask for an IP not found in consul, you get SERVFAIL error because the code asks for the recursors function.. even if there are none in config...

#### Reproduction Steps (issue 2)

Steps to reproduce this issue, eg:

1. Create a cluster with n client nodes n and n server nodes and have an external dns to forward reverse dns to consul. Consul agent config file does not have recursors in it.
1. Run `dig -x Ip address not known by consul`
1. get SERVFAIL ..

#### Reverse DNS enhancement proposal (issue 2):

Shouldn't Consul answer NXDOMAIN (with some TTL?) in case there are no recursors configured? instead of SERVFAIL?
I assume it should play the DNS authoritative role here...

### Consul info for both Client

Client info

```
consul info
agent:
check_monitors = 3
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = 27de64da
version = 1.10.0
consul:
acl = disabled
known_servers = 5
server = false
runtime:
arch = amd64
cpu_count = 4
goroutines = 1100
max_procs = 4
os = linux
version = go1.16.5
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 5529
failed = 1
health_score = 0
intent_queue = 0
left = 0
member_time = 530814
members = 1159
query_queue = 0
query_time = 5962
```

### Operating system and Environment details

OS Linux, Ubuntu 18.04

### Workaround:

We made use of Powerdns minimum-ttl-override (i.e to 5 sec) to lower the load on consul agent.

Best regards,
Marc

Contributor guide

Open the contributing guide

Research direction

Start by reproducing both reverse-DNS cases with the documented Consul setup and `dig -x`: a known address that returns TTL 0 and an unknown address that returns SERVFAIL without recursors configured. Review the reverse-DNS handling and Consul DNS settings, then verify behavior against the proposed Node or Service TTL and NXDOMAIN outcomes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.