hashicorp / hashicorp/consul

Disabling Maintenance mode does not trigger an event

Open
#11,330 8 comments 0 reactions 1 assignee Claimed by @jakubgs View on GitHub
Hacktoberfest type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

I was looking at disabling alerts in `consul-alerts` when host is in maintenance mode: https://github.com/AcalephStorage/consul-alerts/issues/213

And I found out that while `consul-alerts` does create an entry in the KV store for `_node_maintenance` service check when maintenance mode is disabled, it does NOT update that entry once maintenance mode is disabled:

![image](https://user-images.githubusercontent.com/2212681/137499477-b417c01c-f758-4ab6-b0c0-40f66304c155.png)

So I tried running `consul watch` myself same [as the `consul-alerts` service does](https://github.com/AcalephStorage/consul-alerts/blob/v0.6.0/watchers.go#L15-L25) and I found out that disabling maintenance mode does not trigger an event:
```
> consul watch -type checks cat | jq '.[] | select(.CheckID == "_node_maintenance")'
{
"Node": "node-01.example.org",
"CheckID": "_node_maintenance",
"Name": "Node Maintenance Mode",
"Status": "critical",
"Notes": "TEST",
"Output": "",
"ServiceID": "",
"ServiceName": "",
"ServiceTags": [],
"Type": "maintenance",
"Definition": {
"Interval": "0s",
"Timeout": "0s",
"DeregisterCriticalServiceAfter": "0s",
"HTTP": "",
"Header": null,
"Method": "",
"Body": "",
"TLSServerName": "",
"TLSSkipVerify": false,
"TCP": ""
},
"CreateIndex": 67303625,
"ModifyIndex": 67303625
}
```
I receive one event with `Status` set to `critical`, but no recovery to `passing`, since the `_node_maintenance` check is removed.

For this reason `consul-alerts` never updates its stage of `_node_maintenance` check.

#### Reproduction Steps

1. Use version `1.10.1`
1. Run `consul watch -type checks cat`
1. See that no even is triggered for disabling of maintenance mode

### Operating system and Environment details

Ubuntu 20.04.3, `x86_64`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.