hashicorp / hashicorp/consul

Persistent diff in `TaggedAddresses` seems to cause perpetual service resync

Open
#18,834 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

After deploying Consul Connect across the ~5k services in our Nomad cluster, we noticed that the commit rate to the Consul Catalog unexpected jumped from ~0.1/s to ~50/s, all service registrations.

Using `consul monitor -log-level debug` on an agent showed that the `-sidecar-proxy` tasks were repeatedly syncing on every agent back to the main catalog (which would in turn spam a ton of logs about the health checks updating) during the anti-entropy checks.

There was no expected change to the service to cause this, so I dumped the output of `/catalog/node-services/` on a leader and compared it against `/agent/services` on the agent with the Nomad job.

Following the code in the [`IsSame` implementation for services](https://github.com/hashicorp/consul/blob/main/agent/structs/structs.go#L1687) the only thing I can see causing a diff is the `TaggedAddresses` struct, which contains an extra `consul-virtual` address only in the catalog but not on the agent. This has not been added by me anywhere, and seems to be injected by Consul itself?

Service output from the catalog:
```
{
"Kind": "connect-proxy",
"ID": "_nomad-task-fee49b22-[shortened]",
"Service": "[shortened]-sidecar-proxy",
"Tags": [],
"Address": "10.1.110.254",
"TaggedAddresses": {
"consul-virtual": {
"Address": "240.0.36.17",
"Port": 27124
},
"lan_ipv4": {
"Address": "10.1.110.254",
"Port": 27124
},
"wan_ipv4": {
"Address": "10.1.110.254",
"Port": 27124
}
},
"Meta": {
"external-source": "nomad"
},
"Port": 27124,
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"Proxy": {
"DestinationServiceName": "[removed]",
"DestinationServiceID": "_nomad-task-fee49b22-[shortened]",
"LocalServiceAddress": "127.0.0.1",
"LocalServicePort": 28275,
"Mode": "",
"Config": {
"bind_address": "0.0.0.0",
"bind_port": 27124,
"envoy_stats_tags": [
"nomad.alloc_id=fee49b22-[shortened]",
"nomad.group=[removed]",
"nomad.job=[removed]",
"nomad.namespace=default"
]
},
"Upstreams": [
{
"DestinationType": "service",
"DestinationName": "[removed]",
"Datacenter": "",
"LocalBindPort": 54320,
"Config": {
"protocol": "grpc"
},
"MeshGateway": {}
}
],
"MeshGateway": {},
"Expose": {}
},
"Connect": {},
"PeerName": "",
"CreateIndex": 46322912,
"ModifyIndex": 46322912
}
```

Output from the agent:
```
"_nomad-task-fee49b22-[shortened]": {
"Kind": "connect-proxy",
"ID": "_nomad-task-fee49b22-[shortened]",
"Service": "[shortened]-sidecar-proxy",
"Tags": [],
"Meta": {
"external-source": "nomad"
},
"Port": 27124,
"Address": "10.1.110.254",
"TaggedAddresses": {
"lan_ipv4": {
"Address": "10.1.110.254",
"Port": 27124
},
"wan_ipv4": {
"Address": "10.1.110.254",
"Port": 27124
}
},
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"Proxy": {
"DestinationServiceName": "[removed]",
"DestinationServiceID": "_nomad-task-fee49b22-[shortened]",
"LocalServiceAddress": "127.0.0.1",
"LocalServicePort": 28275,
"Config": {
"bind_address": "0.0.0.0",
"bind_port": 27124,
"envoy_stats_tags": [
"nomad.alloc_id=fee49b22-[shortened]",
"nomad.group=[removed]",
"nomad.job=[removed]",
"nomad.namespace=default"
]
},
"Upstreams": [
{
"DestinationType": "service",
"DestinationName": "[removed]",
"LocalBindPort": 54320,
"Config": {
"protocol": "grpc"
},
"MeshGateway": {}
}
],
"MeshGateway": {},
"Expose": {}
},
"Datacenter": "us-east-1"
}
```
---

#### Reproduction Steps

The only thing that might be unusual about my set up is that the topology is we run some ~5000 standalone services (i.e. no one talks to them over the mesh) that all talk to the same upstream service. Otherwise, this is a very basic/standard Consul Connect setup similar to the dashboard/counter example.

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 227
services = 228
build:
prerelease =
revision = e0ab4d29
version = 1.16.1
version_metadata =
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = arm64
cpu_count = 8
goroutines = 3553
max_procs = 8
os = linux
version = go1.20.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 146
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 214378
members = 97
query_queue = 0
query_time = 1
```

```
data_dir = "/cvx/data/consul"
retry_join = ["provider=aws tag_key=nomad-role tag_value=server"]
leave_on_terminate = true
ui_config {
enabled = true
}
limits {
http_max_conns_per_client = 400
}
telemetry {
disable_hostname = true
prometheus_retention_time = "1h"
}
connect {
enabled = true
}
ports {
grpc = 8502
grpc_tls = 8503
}
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = e0ab4d29
version = 1.16.1
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.1.132.43:8300
server = true
raft:
applied_index = 46634577
commit_index = 46634577
fsm_pending = 0
last_contact = 28.437339ms
last_log_index = 46634578
last_log_term = 13
last_snapshot_index = 46633924
last_snapshot_term = 13
latest_configuration = [{Suffrage:Voter ID:60cecced-0a67-8ca3-28ed-1e12d0d110dd Address:10.1.132.43:8300} {Suffrage:Voter ID:8f9b18f2-32b4-b31e-5495-fae11fd8785f Address:10.1.61.239:8300} {Suffrage:Voter ID:6c754ad7-d26e-d60f-6595-1e50468f32b9 Address:10.1.107.51:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 13
runtime:
arch = arm64
cpu_count = 8
goroutines = 42140
max_procs = 8
os = linux
version = go1.20.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 146
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 214378
members = 97
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 19181
members = 3
query_queue = 0
query_time = 1
```

```
(config.hcl is same as client)
```

### Operating system and Environment details

Ubuntu 20.04.6, ARM64

Contributor guide

Open the contributing guide

Research direction

Start with the IsSame implementation in agent/structs/structs.go and compare how TaggedAddresses are represented by /catalog/node-services/ and /agent/services. Reproduce the anti-entropy behavior using the Connect and Nomad setup described, then verify that unchanged service registrations no longer resync repeatedly or create catalog commit and health-check log spam.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.