hashicorp / hashicorp/consul

Can not collect Prometheus metrics from a client agent

Open
#7,684 0 comments 0 reactions 0 assignees View on GitHub
theme/api theme/telemetry
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

### Overview of the Issue

Can not collect Prometheus metrics from a client agent. I get the following error:

```
$ curl -sS http://localhost:8500/v1/agent/metrics?format=prometheus
Prometheus is not enabled since its retention time is not positive
```

I would expect the client to present the metrics collected from a server. IMHO i should be able to collect the metrics from a client as well as a server. In that way Prometheus needs to talk to it's local agent instead of hard coded server IP that can change any time in a highly dynamic cloud environment.

**NOTE**: The same command runs successfully on the servers

#### Reproduction Steps

Enable prometheus metrics in the server agent config:

```
"telemetry": {
"prometheus_retention_time": "7200s",
"disable_hostname": true
},
```
as per documentation https://www.consul.io/docs/agent/options.html#telemetry and run:

```
$ curl -sS http://localhost:8500/v1/agent/metrics?format=prometheus
```

on a client agent.

**NOTE**: Enabling the telemetry on the client agent does not make any difference

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 1
check_ttls = 0
checks = 1
services = 1
build:
prerelease =
revision = 9ea1a204
version = 1.7.2
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 82
max_procs = 2
os = linux
version = go1.13.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 111
members = 17
query_queue = 0
query_time = 1
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 9ea1a204
version = 1.7.2
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.222.4.54:8300
server = true
raft:
applied_index = 18179
commit_index = 18179
fsm_pending = 0
last_contact = 26.722134ms
last_log_index = 18179
last_log_term = 2
last_snapshot_index = 16390
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:edeefb5a-9b98-a2a1-7520-c4873cfdac9c Address:10.222.3.97:8300} {Suffrage:Voter ID:4bfa7045-6db3-b5a5-8095-64a31eba8fff Address:10.222.4.54:8300} {Suffrage:Voter ID:9d130e1e-3739-afee-9f01-168926a03d6e Address:10.222.5.83:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 2
runtime:
arch = amd64
cpu_count = 2
goroutines = 113
max_procs = 2
os = linux
version = go1.13.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 111
members = 17
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 4
members = 3
query_queue = 0
query_time = 1
```

### Operating system and Environment details

```
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
```

### Log Fragments

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.