Incorrect metric consul_memberlist_node_instances
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
When one of nodes going down (server crash) it stays in 'dead' state in the consul interface for 72 hours (as expected).
But the prometheus metric of the consul server `consul_memberlist_node_instances{node_state="dead"}` stays 1 only for a couple of minutes.
Metric `consul_memberlist_node_instances{node_state="alive"}` goes down by 1 as expected.
---
#### Reproduction Steps
2 machines running:
- Consul server
- Consul client
Power down Consul client machine
Monitor
http://consul-server.lan:8500/ui/dc1/nodes
and
http://consul-server.lan:8500/v1/agent/metrics?format=prometheus
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 0
check_ttls = 6
checks = 9
services = 9
build:
prerelease =
revision = 192df66a
version = 1.16.0
version_metadata =
consul:
acl = disabled
known_servers = 5
server = false
runtime:
arch = amd64
cpu_count = 4
goroutines = 100
max_procs = 4
os = linux
version = go1.20.4
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 120
failed = 1
health_score = 0
intent_queue = 0
left = 0
member_time = 16066
members = 61
query_queue = 0
query_time = 2
```
```
data_dir = "/opt/consul"
```
Server info
```
agent:
check_monitors = 0
check_ttls = 5
checks = 9
services = 9
build:
prerelease =
revision = 192df66a
version = 1.16.0
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 192.168.0.1:8300
server = true
raft:
applied_index = 3628061
commit_index = 3628061
fsm_pending = 0
last_contact = 0
last_log_index = 3628061
last_log_term = 22
last_snapshot_index = 3623276
last_snapshot_term = 22
latest_configuration = []
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 22
runtime:
arch = amd64
cpu_count = 4
goroutines = 941
max_procs = 4
os = linux
version = go1.20.4
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 120
failed = 1
health_score = 0
intent_queue = 0
left = 0
member_time = 16066
members = 61
query_queue = 0
query_time = 2
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 215
members = 5
query_queue = 0
query_time = 1
```
```
data_dir = "/opt/consul"
```
### Operating system and Environment details
### Log Fragments
Contributor guide
Assessment
This issue has not been assessed yet.