hashicorp / hashicorp/consul

Keep recursor TTL

Open
#14,703 0 comments 0 reactions 0 assignees View on GitHub
theme/dns theme/operator-usability
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

Hi, we are experiencing a problem when using external services registered in consul. Our upstream DNS resolver is being flooded with queries.

We're using consul 1.2.4 and we have DNS forward queries to consul and also dnsmasq as a recursor for consul.
We have several external services registered so when using those services the external URL is transparent for end services.

Doing some debug we saw the following (TTL is both 0 for the CNAME and A records):

```
# dig external.service.consul

; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> external.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41233
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;external.service.consul. IN A

;; ANSWER SECTION:
external.service.consul. 0 IN CNAME external-domain.mycustom.domain.
external-domain.mycustom.domain. 0 IN A 10.111.23.220

;; Query time: 2 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Sep 22 06:13:34 UTC 2022
;; MSG SIZE rcvd: 118
```

Consul overrides the recursor TTL value to 0 so the domain is not being cached by dnsmasq and is always queried to the upstream DNS (in this case is the 10.48.33.33).

```
Sep 22 06:08:51 dnsmasq[403749]: 404 127.0.0.1/39632 query[A] external.service.consul from 127.0.0.1
Sep 22 06:08:51 dnsmasq[403749]: 404 127.0.0.1/39632 forwarded external.service.consul to 127.0.0.1
Sep 22 06:08:51 dnsmasq[403749]: 405 10.171.0.24/44607 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:08:51 dnsmasq[403749]: 405 10.171.0.24/44607 forwarded external-domain.mycustom.domain to 10.48.33.33
Sep 22 06:08:51 dnsmasq[403749]: 405 10.171.0.24/44607 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:51 dnsmasq[403749]: 404 127.0.0.1/39632 reply external.service.consul is
Sep 22 06:08:51 dnsmasq[403749]: 404 127.0.0.1/39632 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:51 dnsmasq[403749]: 406 127.0.0.1/48987 query[A] external.service.consul from 127.0.0.1
Sep 22 06:08:51 dnsmasq[403749]: 406 127.0.0.1/48987 forwarded external.service.consul to 127.0.0.1
Sep 22 06:08:51 dnsmasq[403749]: 407 10.171.0.24/49355 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:08:51 dnsmasq[403749]: 407 10.171.0.24/49355 forwarded external-domain.mycustom.domain to 10.48.33.33
Sep 22 06:08:51 dnsmasq[403749]: 407 10.171.0.24/49355 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:51 dnsmasq[403749]: 406 127.0.0.1/48987 reply external.service.consul is
Sep 22 06:08:51 dnsmasq[403749]: 406 127.0.0.1/48987 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:51 dnsmasq[403749]: 408 127.0.0.1/56239 query[A] external.service.consul from 127.0.0.1
Sep 22 06:08:51 dnsmasq[403749]: 408 127.0.0.1/56239 forwarded external.service.consul to 127.0.0.1
Sep 22 06:08:51 dnsmasq[403749]: 409 10.171.0.24/45665 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:08:51 dnsmasq[403749]: 409 10.171.0.24/45665 forwarded external-domain.mycustom.domain to 10.48.33.33
Sep 22 06:08:51 dnsmasq[403749]: 409 10.171.0.24/45665 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:51 dnsmasq[403749]: 408 127.0.0.1/56239 reply external.service.consul is
Sep 22 06:08:51 dnsmasq[403749]: 408 127.0.0.1/56239 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:51 dnsmasq[403749]: 410 127.0.0.1/46164 query[A] external.service.consul from 127.0.0.1
Sep 22 06:08:51 dnsmasq[403749]: 410 127.0.0.1/46164 forwarded external.service.consul to 127.0.0.1
Sep 22 06:08:51 dnsmasq[403749]: 411 10.171.0.24/52671 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:08:51 dnsmasq[403749]: 411 10.171.0.24/52671 forwarded external-domain.mycustom.domain to 10.48.33.33
Sep 22 06:08:51 dnsmasq[403749]: 411 10.171.0.24/52671 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:51 dnsmasq[403749]: 410 127.0.0.1/46164 reply external.service.consul is
Sep 22 06:08:51 dnsmasq[403749]: 410 127.0.0.1/46164 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:51 dnsmasq[403749]: 412 127.0.0.1/54850 query[A] external.service.consul from 127.0.0.1
Sep 22 06:08:51 dnsmasq[403749]: 412 127.0.0.1/54850 forwarded external.service.consul to 127.0.0.1
Sep 22 06:08:51 dnsmasq[403749]: 413 10.171.0.24/36028 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:08:51 dnsmasq[403749]: 413 10.171.0.24/36028 forwarded external-domain.mycustom.domain to 10.48.33.33
Sep 22 06:08:51 dnsmasq[403749]: 413 10.171.0.24/36028 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:51 dnsmasq[403749]: 412 127.0.0.1/54850 reply external.service.consul is
Sep 22 06:08:51 dnsmasq[403749]: 412 127.0.0.1/54850 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:52 dnsmasq[403749]: 414 127.0.0.1/44773 query[A] external.service.consul from 127.0.0.1
Sep 22 06:08:52 dnsmasq[403749]: 414 127.0.0.1/44773 forwarded external.service.consul to 127.0.0.1
Sep 22 06:08:52 dnsmasq[403749]: 415 10.171.0.24/51162 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:08:52 dnsmasq[403749]: 415 10.171.0.24/51162 forwarded external-domain.mycustom.domain to 10.48.33.33
Sep 22 06:08:52 dnsmasq[403749]: 415 10.171.0.24/51162 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:08:52 dnsmasq[403749]: 414 127.0.0.1/44773 reply external.service.consul is
Sep 22 06:08:52 dnsmasq[403749]: 414 127.0.0.1/44773 reply external-domain.mycustom.domain is 10.111.23.220
```

I know we can set a `service_ttl` in the configuration but that's per service and we only wan't to keep the ttl of registered external services and we would need to change the configuration file of all clients every time we add a external service.

I think it would be useful to have an option to keep the TTL returned by the recursor DNS when querying external services.

If this option is configured a resolution would behave in the following way:

```
# dig external.service.consul

; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> external.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34669
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;external.service.consul. IN A

;; ANSWER SECTION:
external.service.consul. 0 IN CNAME external-domain.mycustom.domain.
external-domain.mycustom.domain. 7200 IN A 10.111.23.220

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Sep 22 06:23:54 UTC 2022
;; MSG SIZE rcvd: 118

#
# dig external.service.consul

; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> external.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60117
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;external.service.consul. IN A

;; ANSWER SECTION:
external.service.consul. 0 IN CNAME external-domain.mycustom.domain.
external-domain.mycustom.domain. 7193 IN A 10.111.23.220

;; Query time: 2 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Sep 22 06:24:01 UTC 2022
;; MSG SIZE rcvd: 118
```

The TTL of the external resolution would be kept in the A record. So it's being cached by dnsmasq:

```
Sep 22 06:23:54 dnsmasq[403749]: 522 127.0.0.1/56005 query[A] external.service.consul from 127.0.0.1
Sep 22 06:23:54 dnsmasq[403749]: 522 127.0.0.1/56005 forwarded external.service.consul to 127.0.0.1
Sep 22 06:23:54 dnsmasq[403749]: 523 10.171.0.24/44682 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:23:54 dnsmasq[403749]: 523 10.171.0.24/44682 forwarded external-domain.mycustom.domain to 10.48.33.33
Sep 22 06:23:54 dnsmasq[403749]: 523 10.171.0.24/44682 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:23:54 dnsmasq[403749]: 522 127.0.0.1/56005 reply external.service.consul is
Sep 22 06:23:54 dnsmasq[403749]: 522 127.0.0.1/56005 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:24:01 dnsmasq[403749]: 524 127.0.0.1/53952 query[A] external.service.consul from 127.0.0.1
Sep 22 06:24:01 dnsmasq[403749]: 524 127.0.0.1/53952 forwarded external.service.consul to 127.0.0.1
Sep 22 06:24:01 dnsmasq[403749]: 525 10.171.0.24/35386 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:24:01 dnsmasq[403749]: 525 10.171.0.24/35386 cached external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:24:01 dnsmasq[403749]: 524 127.0.0.1/53952 reply external.service.consul is
Sep 22 06:24:01 dnsmasq[403749]: 524 127.0.0.1/53952 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:24:02 dnsmasq[403749]: 526 127.0.0.1/52331 query[A] external.service.consul from 127.0.0.1
Sep 22 06:24:02 dnsmasq[403749]: 526 127.0.0.1/52331 forwarded external.service.consul to 127.0.0.1
Sep 22 06:24:02 dnsmasq[403749]: 527 10.171.0.24/37448 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:24:02 dnsmasq[403749]: 527 10.171.0.24/37448 cached external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:24:02 dnsmasq[403749]: 526 127.0.0.1/52331 reply external.service.consul is
Sep 22 06:24:02 dnsmasq[403749]: 526 127.0.0.1/52331 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:24:03 dnsmasq[403749]: 528 127.0.0.1/54975 query[A] external.service.consul from 127.0.0.1
Sep 22 06:24:03 dnsmasq[403749]: 528 127.0.0.1/54975 forwarded external.service.consul to 127.0.0.1
Sep 22 06:24:03 dnsmasq[403749]: 529 10.171.0.24/42744 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:24:03 dnsmasq[403749]: 529 10.171.0.24/42744 cached external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:24:03 dnsmasq[403749]: 528 127.0.0.1/54975 reply external.service.consul is
Sep 22 06:24:03 dnsmasq[403749]: 528 127.0.0.1/54975 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:24:05 dnsmasq[403749]: 530 127.0.0.1/51257 query[A] external.service.consul from 127.0.0.1
Sep 22 06:24:05 dnsmasq[403749]: 530 127.0.0.1/51257 forwarded external.service.consul to 127.0.0.1
Sep 22 06:24:05 dnsmasq[403749]: 531 10.171.0.24/38411 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:24:05 dnsmasq[403749]: 531 10.171.0.24/38411 cached external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:24:05 dnsmasq[403749]: 530 127.0.0.1/51257 reply external.service.consul is
Sep 22 06:24:05 dnsmasq[403749]: 530 127.0.0.1/51257 reply external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:24:06 dnsmasq[403749]: 532 127.0.0.1/58263 query[A] external.service.consul from 127.0.0.1
Sep 22 06:24:06 dnsmasq[403749]: 532 127.0.0.1/58263 forwarded external.service.consul to 127.0.0.1
Sep 22 06:24:06 dnsmasq[403749]: 533 10.171.0.24/33264 query[A] external-domain.mycustom.domain from 10.171.0.24
Sep 22 06:24:06 dnsmasq[403749]: 533 10.171.0.24/33264 cached external-domain.mycustom.domain is 10.111.23.220
Sep 22 06:24:06 dnsmasq[403749]: 532 127.0.0.1/58263 reply external.service.consul is
Sep 22 06:24:06 dnsmasq[403749]: 532 127.0.0.1/58263 reply external-domain.mycustom.domain is 10.111.23.220
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing Consul's DNS handling for external services and how recursor responses are converted into CNAME and A records. The change is done when a configurable behavior preserves the recursor-provided TTL for external-service A records, allowing dnsmasq to cache them without changing each service's configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.