Persistent diff in `TaggedAddresses` seems to cause perpetual service resync
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
After deploying Consul Connect across the ~5k services in our Nomad cluster, we noticed that the commit rate to the Consul Catalog unexpected jumped from ~0.1/s to ~50/s, all service registrations.
Using `consul monitor -log-level debug` on an agent showed that the `-sidecar-proxy` tasks were repeatedly syncing on every agent back to the main catalog (which would in turn spam a ton of logs about the health checks updating) during the anti-entropy checks.
There was no expected change to the service to cause this, so I dumped the output of `/catalog/node-services/` on a leader and compared it against `/agent/services` on the agent with the Nomad job.
Following the code in the [`IsSame` implementation for services](https://github.com/hashicorp/consul/blob/main/agent/structs/structs.go#L1687) the only thing I can see causing a diff is the `TaggedAddresses` struct, which contains an extra `consul-virtual` address only in the catalog but not on the agent. This has not been added by me anywhere, and seems to be injected by Consul itself?
Service output from the catalog:
```
{
"Kind": "connect-proxy",
"ID": "_nomad-task-fee49b22-[shortened]",
"Service": "[shortened]-sidecar-proxy",
"Tags": [],
"Address": "10.1.110.254",
"TaggedAddresses": {
"consul-virtual": {
"Address": "240.0.36.17",
"Port": 27124
},
"lan_ipv4": {
"Address": "10.1.110.254",
"Port": 27124
},
"wan_ipv4": {
"Address": "10.1.110.254",
"Port": 27124
}
},
"Meta": {
"external-source": "nomad"
},
"Port": 27124,
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"Proxy": {
"DestinationServiceName": "[removed]",
"DestinationServiceID": "_nomad-task-fee49b22-[shortened]",
"LocalServiceAddress": "127.0.0.1",
"LocalServicePort": 28275,
"Mode": "",
"Config": {
"bind_address": "0.0.0.0",
"bind_port": 27124,
"envoy_stats_tags": [
"nomad.alloc_id=fee49b22-[shortened]",
"nomad.group=[removed]",
"nomad.job=[removed]",
"nomad.namespace=default"
]
},
"Upstreams": [
{
"DestinationType": "service",
"DestinationName": "[removed]",
"Datacenter": "",
"LocalBindPort": 54320,
"Config": {
"protocol": "grpc"
},
"MeshGateway": {}
}
],
"MeshGateway": {},
"Expose": {}
},
"Connect": {},
"PeerName": "",
"CreateIndex": 46322912,
"ModifyIndex": 46322912
}
```
Output from the agent:
```
"_nomad-task-fee49b22-[shortened]": {
"Kind": "connect-proxy",
"ID": "_nomad-task-fee49b22-[shortened]",
"Service": "[shortened]-sidecar-proxy",
"Tags": [],
"Meta": {
"external-source": "nomad"
},
"Port": 27124,
"Address": "10.1.110.254",
"TaggedAddresses": {
"lan_ipv4": {
"Address": "10.1.110.254",
"Port": 27124
},
"wan_ipv4": {
"Address": "10.1.110.254",
"Port": 27124
}
},
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"Proxy": {
"DestinationServiceName": "[removed]",
"DestinationServiceID": "_nomad-task-fee49b22-[shortened]",
"LocalServiceAddress": "127.0.0.1",
"LocalServicePort": 28275,
"Config": {
"bind_address": "0.0.0.0",
"bind_port": 27124,
"envoy_stats_tags": [
"nomad.alloc_id=fee49b22-[shortened]",
"nomad.group=[removed]",
"nomad.job=[removed]",
"nomad.namespace=default"
]
},
"Upstreams": [
{
"DestinationType": "service",
"DestinationName": "[removed]",
"LocalBindPort": 54320,
"Config": {
"protocol": "grpc"
},
"MeshGateway": {}
}
],
"MeshGateway": {},
"Expose": {}
},
"Datacenter": "us-east-1"
}
```
---
#### Reproduction Steps
The only thing that might be unusual about my set up is that the topology is we run some ~5000 standalone services (i.e. no one talks to them over the mesh) that all talk to the same upstream service. Otherwise, this is a very basic/standard Consul Connect setup similar to the dashboard/counter example.
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 227
services = 228
build:
prerelease =
revision = e0ab4d29
version = 1.16.1
version_metadata =
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = arm64
cpu_count = 8
goroutines = 3553
max_procs = 8
os = linux
version = go1.20.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 146
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 214378
members = 97
query_queue = 0
query_time = 1
```
```
data_dir = "/cvx/data/consul"
retry_join = ["provider=aws tag_key=nomad-role tag_value=server"]
leave_on_terminate = true
ui_config {
enabled = true
}
limits {
http_max_conns_per_client = 400
}
telemetry {
disable_hostname = true
prometheus_retention_time = "1h"
}
connect {
enabled = true
}
ports {
grpc = 8502
grpc_tls = 8503
}
```
Server info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = e0ab4d29
version = 1.16.1
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.1.132.43:8300
server = true
raft:
applied_index = 46634577
commit_index = 46634577
fsm_pending = 0
last_contact = 28.437339ms
last_log_index = 46634578
last_log_term = 13
last_snapshot_index = 46633924
last_snapshot_term = 13
latest_configuration = [{Suffrage:Voter ID:60cecced-0a67-8ca3-28ed-1e12d0d110dd Address:10.1.132.43:8300} {Suffrage:Voter ID:8f9b18f2-32b4-b31e-5495-fae11fd8785f Address:10.1.61.239:8300} {Suffrage:Voter ID:6c754ad7-d26e-d60f-6595-1e50468f32b9 Address:10.1.107.51:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 13
runtime:
arch = arm64
cpu_count = 8
goroutines = 42140
max_procs = 8
os = linux
version = go1.20.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 146
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 214378
members = 97
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 19181
members = 3
query_queue = 0
query_time = 1
```
```
(config.hcl is same as client)
```
### Operating system and Environment details
Ubuntu 20.04.6, ARM64
Contributor guide
Research direction
Start with the IsSame implementation in agent/structs/structs.go and compare how TaggedAddresses are represented by /catalog/node-services/ and /agent/services. Reproduce the anti-entropy behavior using the Connect and Nomad setup described, then verify that unchanged service registrations no longer resync repeatedly or create catalog commit and health-check log spam.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100