Disabling Maintenance mode does not trigger an event
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
I was looking at disabling alerts in `consul-alerts` when host is in maintenance mode: https://github.com/AcalephStorage/consul-alerts/issues/213
And I found out that while `consul-alerts` does create an entry in the KV store for `_node_maintenance` service check when maintenance mode is disabled, it does NOT update that entry once maintenance mode is disabled:

So I tried running `consul watch` myself same [as the `consul-alerts` service does](https://github.com/AcalephStorage/consul-alerts/blob/v0.6.0/watchers.go#L15-L25) and I found out that disabling maintenance mode does not trigger an event:
```
> consul watch -type checks cat | jq '.[] | select(.CheckID == "_node_maintenance")'
{
"Node": "node-01.example.org",
"CheckID": "_node_maintenance",
"Name": "Node Maintenance Mode",
"Status": "critical",
"Notes": "TEST",
"Output": "",
"ServiceID": "",
"ServiceName": "",
"ServiceTags": [],
"Type": "maintenance",
"Definition": {
"Interval": "0s",
"Timeout": "0s",
"DeregisterCriticalServiceAfter": "0s",
"HTTP": "",
"Header": null,
"Method": "",
"Body": "",
"TLSServerName": "",
"TLSSkipVerify": false,
"TCP": ""
},
"CreateIndex": 67303625,
"ModifyIndex": 67303625
}
```
I receive one event with `Status` set to `critical`, but no recovery to `passing`, since the `_node_maintenance` check is removed.
For this reason `consul-alerts` never updates its stage of `_node_maintenance` check.
#### Reproduction Steps
1. Use version `1.10.1`
1. Run `consul watch -type checks cat`
1. See that no even is triggered for disabling of maintenance mode
### Operating system and Environment details
Ubuntu 20.04.3, `x86_64`
Contributor guide
Assessment
This issue has not been assessed yet.