hashicorp / hashicorp/consul

Consul aliased check doesn't clear after node maintenance

Open
#6,797 3 comments 1 reaction 0 assignees View on GitHub
needs-investigation type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

I recently performed a rolling update that took advantage of the node-level `consul maint -enable`. After clearing the maintenance check, I noticed the nodes have their aliased service checks broken: they are flagged as critical, and their output is that of an _unrelated_ node check (which is passing anyway!). Watching logs, Consul doesn't seem to mention or update the invalid service.
I've found a restart of the agent may clear it. I've had to restart the agent repeatedly to get the correct (passing status) behavior. I have one or two where restarting the agent is not addressing the problem.

Notice how the output is about the cloudinit node check and not the elasticsearch service it's alias'd to (see conf example later on):
```
$ curl http://localhost:8500/v1/agent/checks | python -m json.tool
...
"service:elk-monitoring": {
"CheckID": "service:elk-monitoring",
"CreateIndex": 0,
"Definition": {},
"ModifyIndex": 0,
"Name": "monitoring alias",
"Node": "ip-10-111-24-239.us-east-2.compute.internal",
"Notes": "",
"Output": "Aliased check \"cloudinit boot finished\" failing: ",
"ServiceID": "elk-monitoring",
"ServiceName": "elk-monitoring",
"ServiceTags": [
"warm"
],
"Status": "critical"
}
```

If there's any more information I could provide that could help identify where the issue lies I'd be happy to provide it. Thank you!
#### Reproduction Steps

Steps to reproduce this issue, eg:

1. Run `consul maint -enable -reason="test"`
1. Run `consul maint -disable`
1. Check status

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 2
check_ttls = 0
checks = 5
services = 4
build:
prerelease =
revision = 567e41ff
version = 1.4.3
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 8
goroutines = 53
max_procs = 8
os = linux
version = go1.11.4
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 14
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 11137
members = 148
query_queue = 0
query_time = 1
```

Server info

```
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 1
build:
prerelease =
revision = ea5210a3
version = 1.4.4
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 10.111.0.42:8300
server = true
raft:
applied_index = 1458814
commit_index = 1458814
fsm_pending = 0
last_contact = 0
last_log_index = 1458814
last_log_term = 3
last_snapshot_index = 1442639
last_snapshot_term = 3
latest_configuration = [{Suffrage:Voter ID:24520abe-ff70-d114-3ba9-9713ba4c6b6e Address:10.111.0.42:8300} {Suffrage:Voter ID:f5c456dc-29df-6b7c-8f69-93f4dfd71af4 Address:10.111.2.42:8300} {Suffrage:Voter ID:90bc90ad-c256-2281-a550-320f5f8ec16b Address:10.111.1.42:8300}]
latest_configuration_index = 1
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 3
runtime:
arch = amd64
cpu_count = 4
goroutines = 528
max_procs = 4
os = linux
version = go1.11.4
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 3
failed = 0
health_score = 0
intent_queue = 0
left = 3
member_time = 942
members = 124
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 3
query_queue = 0
query_time = 1
```

### Operating system and Environment details

CentOS 7. Consul v1.4.3.
This happens in both our staging and production environments (different capacity consul servers).

```
$ cat /opt/consul/config.d/service-elk-*.hcl
service {
name = "elk-elasticsearch"
port = 9200
tags = ["data", "hot"]
check {
http = "http://localhost:9200/_nodes/_local/stats/process"
interval = "10s"
}
}
service {
name = "elk-elasticsearch-transport"
port = 9300
tags = ["data", "hot"]
check {
tcp = "localhost:9300"
interval = "10s"
}
}
service {
name = "elk-monitoring"
port = 9200
tags = ["hot"]
check {
name = "monitoring alias"
alias_service = "elk-elasticsearch"
}
}

$ cat /opt/consul/config.d/consul.hcl
checks {
name = "cloudinit boot finished"
args = ["/bin/cat", "/var/lib/cloud/instance/boot-finished"]
interval = "10s"
timeout = "1s"
}
```

### Log Fragments

There's nothing I can find of any interest in the logs. If there's specific things I should go dig up, I'd be happy to

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.