Watches being triggered during central config entries changes
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
Similar to https://github.com/hashicorp/consul/issues/7446 but it's happening when applying any change to central config entries using the consul cli (config write, config delete)
We have checks watchers configured in consul servers and clients:
- Servers watch: on trigger they check what nodes are in critical status (api endpoint `/v1/health/state/critical`) and if they find any they will mark the node as unhealthy so it can be replaced by the autoscaling group
- Client watch: on trigger they check all their local checks (api endpoint `/v1/agent/checks?filter=Status!=passing`) and if any is in non passing status they mark themselves as an unhealthy node so it can be replaced by the autoscaling group
Every time we apply a new config entry with the consul cli (consul config write or consul config delete) the watchers are triggered in servers and clients, and the watchers scripts find checks in critical status so leading to the replacement of all of our nodes.
If we don't make any consul config change, it works fine, the unhealthy nodes are replaced when they need to, new nodes join without problems, etc.
#### Reproduction Steps
1. Create a cluster similar to the described further in [Operating system and Environment details](#operating-system-and-environment-details)
2. Once all services are up and running with all sidecar proxies and services with checks passing, apply any change to config with consul config cli (you can add a new service-defaults entry for example, even for a non existent service)
3. The checks watches from the server and the clients will be triggered and some checks from each node will be in critical status
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 2
build:
prerelease =
revision =
version = 1.15.1
version_metadata =
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 98
max_procs = 2
os = linux
version = go1.20.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 34
failed = 0
health_score = 0
intent_queue = 0
left = 5
member_time = 2380157
members = 84
query_queue = 0
query_time = 1
```
Server info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision =
version = 1.15.1
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 172.31.50.254:8300
server = true
raft:
applied_index = 23759419
commit_index = 23759419
fsm_pending = 0
last_contact = 7.852926ms
last_log_index = 23759419
last_log_term = 55
last_snapshot_index = 23750571
last_snapshot_term = 55
latest_configuration = [{Suffrage:Voter ID:7e97476a-94dd-783f-0645-a9e78861ae6d Address:172.31.50.254:8300} {Suffrage:Voter ID:858454d3-9a87-48ef-c342-05f954076701 Address:172.31.77.53:8300} {Suffrage:Voter ID:d1d55041-728a-6e70-aee7-56421f986ac1 Address:172.31.61.5:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 55
runtime:
arch = amd64
cpu_count = 2
goroutines = 1191
max_procs = 2
os = linux
version = go1.20.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 34
failed = 0
health_score = 0
intent_queue = 0
left = 8
member_time = 2380163
members = 84
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 25790
members = 3
query_queue = 0
query_time = 1
```
### Operating system and Environment details
3 consul servers and ~80 consul clients
OS: Ubuntu 20.04
Arch: x86_64
Reproduced with consul versions:
- 1.14.4 + envoy 1.24.0
- 1.15.1 + envoy 1.24.2
Central config entries:
- http services: service-defaults + service-router
- grpc services: service-defaults
- proxy-defaults with just `envoy_dogstatsd_url` config
- each client node will have a single service registered with 1 or more upstreams through connect envoy.
### Log Fragments
Contributor guide
Research direction
Reproduce with the consul config write and consul config delete commands after setting up the server and client watches. Inspect how central config entry changes relate to the watch checks at /v1/health/state/critical and /v1/agent/checks?filter=Status!=passing. Done means config entry changes no longer trigger unrelated watches or leave checks in critical status; add regression coverage for the described scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100