hashicorp / hashicorp/consul

Consul DNS metrics are lost

Open
#8,292 0 comments 1 reaction 0 assignees View on GitHub
theme/dns theme/telemetry
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue
All dns queries in our prod environment go through our internal DNS servers. This is also true for all internal consul dns queries.
The DNS slaves are configured to forward all `.consul` queries to the local agent at port 8600
```
zone "consul" IN {
type forward;
forward only;
forwarders { 127.0.0.1 port 8600; };
};
```
For some reason we do not see DNS metrics on the consul server or on our DNS slaves.

A while ago we had one datacenter where the DNS slaves were running with wrong configuration and were forwarding all consul queries to the consul servers. During that time we did see dns metrics at `http://consul-server:8500/v1/agent/metrics`

What needs to be changed so that we can get these metrics? (We would like to keep our current forward setup on the dns slaves)

Consul agent version 1.2.1

### DNS slave - agent config
```
{
"data_dir": "/etc/consul",
"encrypt": "<>",
"disable_remote_exec": true,
"disable_update_check": true,
"enable_script_checks": true,
"log_level": "WARN",
"dns_config": {
"allow_stale": true,
"max_stale": "5s",
"enable_truncate": true
},
"datacenter": "dc1",
"start_join": [
"consul-server1.dc1.company.com",
"consul-server2.dc1.company.com",
"consul-server3.dc1.company.com"
],
"raft_protocol": 3,
"advertise_addr": ""
}
```

### Consul server - agent config
```
{
"data_dir": "/etc/consul",
"encrypt": "<>",
"disable_remote_exec": true,
"disable_update_check": true,
"enable_script_checks": true,
"log_level": "WARN",
"dns_config": {
"allow_stale": true,
"max_stale": "5s",
"enable_truncate": true
},
"datacenter": "dc1",
"start_join": [
"consul-server1.dc1.company.com",
"consul-server2.dc1.company.com",
"consul-server3.dc1.company.com"
],
"raft_protocol": 3,
"server": true,
"bootstrap_expect": 3,
"performance": {
"raft_multiplier": 2
},
"ui": true,
"telemetry": {
"prometheus_retention_time": "10m",
"metrics_prefix": "consul_dc1"
},
"client_addr": "0.0.0.0",
"advertise_addr": "local_IP"
}
```

Contributor guide

Open the contributing guide

Research direction

Start at the Consul agent's /v1/agent/metrics endpoint and trace how DNS queries forwarded to 127.0.0.1:8600 are accounted for. Compare the DNS slave and server configurations shown in the report; done means forwarded .consul queries produce DNS metrics without changing the forwarding setup. No source file or test is identified in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus
Domain
networking, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.