Reverse DNS fixed TTL to 0 (hard-coded), or always returning SERVFAIL
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
Hi everyone,
I would like to share some reverse DNS behaviours that we faced, and to open discussions about some enhancement on consul reverse dns behaviour.
I put 2 issues in one ticket, but it is releated to the same code part
####The set-up
1. Few Powerdns recursors forwarding direct resolution and reverse dns to consul agent following [https://learn.hashicorp.com/tutorials/consul/dns-forwarding](https://learn.hashicorp.com/tutorials/consul/dns-forwarding)
2. About 1000 nodes on DC, 500 services...
3. No upstream recursors [https://www.consul.io/docs/agent/options#recursors](https://www.consul.io/docs/agent/options#recursors), meaning I consider Consul as authoritative on both his domain, and the reverse-dns ips allocated...
With nodes, and services growing, we start to see some higher CPU + Network usage coming from Consul agent on DNS server.
#### Overview of the (Issue 1)
After analysis, we observed :
Reverse-dns answers are always with 0 TTL. (hard-coded)
So even if we could play with DNS cache (some TTLs are changed for nodes, services in consul agent config...) it cannot be the case on reverse dns.
#### Reproduction Steps (issue 1)
Steps to reproduce this issue, eg:
1. Create a cluster with n client nodes n and n server nodes, and have an external dns to forward reverse dns to consul
1. Run `dig -x Ip address`
1. TTL is to 0
#### suggested behaviour:
Would it be possible to return Node TTL (in case a Node is found) or Service TTL in case a service is returned ? to be able to use caching mecanisms to reduce load on consul agent ?
If neither of these, could it be default minTTL value declared in SOA? or a specific added parameter?
#### Overview of the (Issue 2)
After analysis, we observed :
If you ask for an IP not found in consul, you get SERVFAIL error because the code asks for the recursors function.. even if there are none in config...
#### Reproduction Steps (issue 2)
Steps to reproduce this issue, eg:
1. Create a cluster with n client nodes n and n server nodes and have an external dns to forward reverse dns to consul. Consul agent config file does not have recursors in it.
1. Run `dig -x Ip address not known by consul`
1. get SERVFAIL ..
#### Reverse DNS enhancement proposal (issue 2):
Shouldn't Consul answer NXDOMAIN (with some TTL?) in case there are no recursors configured? instead of SERVFAIL?
I assume it should play the DNS authoritative role here...
### Consul info for both Client
Client info
```
consul info
agent:
check_monitors = 3
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = 27de64da
version = 1.10.0
consul:
acl = disabled
known_servers = 5
server = false
runtime:
arch = amd64
cpu_count = 4
goroutines = 1100
max_procs = 4
os = linux
version = go1.16.5
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 5529
failed = 1
health_score = 0
intent_queue = 0
left = 0
member_time = 530814
members = 1159
query_queue = 0
query_time = 5962
```
### Operating system and Environment details
OS Linux, Ubuntu 18.04
### Workaround:
We made use of Powerdns minimum-ttl-override (i.e to 5 sec) to lower the load on consul agent.
Best regards,
Marc
Contributor guide
Research direction
Start by reproducing both reverse-DNS cases with the documented Consul setup and `dig -x`: a known address that returns TTL 0 and an unknown address that returns SERVFAIL without recursors configured. Review the reverse-DNS handling and Consul DNS settings, then verify behavior against the proposed Node or Service TTL and NXDOMAIN outcomes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100