consul server log is spammed with: EnsureRegistration failed: failed inserting check: Missing service registration
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
We have 3 physical Consul clusters, all have 3 consul servers. A few days ago the servers of one of the clusters (~1000 nodes) started to fill up their logs with the following message:
`EnsureRegistration failed: failed inserting check: Missing service registration`
We checked the services and checks registered on the node and they seem fine (they were not changed)
### Consul info of Server
Client info
```
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 2
build:
prerelease =
revision = 39f93f01
version = 1.2.1
consul:
bootstrap = false
known_datacenters = 3
leader = false
leader_addr = :8300
server = true
raft:
applied_index = 180693997
commit_index = 180693997
fsm_pending = 0
last_contact = 23.143964ms
last_log_index = 180693998
last_log_term = 3317
last_snapshot_index = 180688388
last_snapshot_term = 3317
latest_configuration = [{Suffrage:Voter ID:022db9dc-bfa0-a441-cb0f-3515ff7a13bb Address::8300} {Suffrage:Voter ID:fd6e7081-3d26-2e88-051a-633706f27135 Address::8300} {Suffrage:Voter ID:65d9e50f-4ca6-173a-aa14-40d623abe4b8 Address::8300}]
latest_configuration_index = 180662756
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 3317
runtime:
arch = amd64
cpu_count = 24
goroutines = 32667
max_procs = 24
os = linux
version = go1.10.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 106242
failed = 0
health_score = 0
intent_queue = 0
left = 18
member_time = 153605
members = 1034
query_queue = 0
query_time = 1172
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1890
members = 9
query_queue = 0
query_time = 1
```
### Operating system and Environment details
All servers run Ubuntu16 with latest kernel version. All servers in LAN are connected via 10Gig network.
### Registered services
```
curl http://localhost:8500/v1/agent/services?pretty
{
"consul-service-8500": {
"Kind": "",
"ID": "consul-service-8500",
"Service": "consul-service",
"Tags": [
"hostType-metal",
"servicetype-consul-service",
"owner-ops",
"scrapeport-9107"
],
"Meta": {},
"Port": 8500,
"Address": "",
"EnableTagOverride": false,
"CreateIndex": 0,
"ModifyIndex": 0,
"ProxyDestination": "",
"Connect": null
},
"consul-service-metrics-8500": {
"Kind": "",
"ID": "consul-service-metrics-8500",
"Service": "consul-service-metrics",
"Tags": [
"hostType-metal",
"servicetype-consul-service-metrics",
"owner-ops",
"scrapeport-8500",
"scrapepath-/v1/agent/metrics?format=prometheus"
],
"Meta": {},
"Port": 8500,
"Address": "",
"EnableTagOverride": false,
"CreateIndex": 0,
"ModifyIndex": 0,
"ProxyDestination": "",
"Connect": null
}
}
```
### Checks
```
curl http://localhost:8500/v1/agent/checks?pretty
{
"consul-service-8500": {
"Node": "",
"CheckID": "consul-service-8500",
"Name": "consul-service Health Status",
"Status": "passing",
"Notes": "",
"Output": "",
"ServiceID": "consul-service-8500",
"ServiceName": "consul-service",
"ServiceTags": [
"hostType-metal",
"servicetype-consul-service",
"owner-ops",
"scrapeport-9107"
],
"Definition": {},
"CreateIndex": 0,
"ModifyIndex": 0
},
"consul-service-metrics-8500": {
"Node": "",
"CheckID": "consul-service-metrics-8500",
"Name": "consul-service-metrics Health Status",
"Status": "passing",
"Notes": "",
"Output": "",
"ServiceID": "consul-service-metrics-8500",
"ServiceName": "consul-service-metrics",
"ServiceTags": [
"hostType-metal",
"servicetype-consul-service-metrics",
"owner-ops",
"scrapeport-8500",
"scrapepath-/v1/agent/metrics?format=prometheus"
],
"Definition": {},
"CreateIndex": 0,
"ModifyIndex": 0
}
}
```
### Log Fragments
2020/03/29 10:28:55 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:30:08 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:31:26 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:32:37 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:33:10 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:34:05 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:35:34 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:36:58 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:37:57 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:39:20 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:40:50 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:42:06 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:42:55 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:43:51 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:44:28 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:44:44 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:46:06 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:47:20 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:48:03 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:48:20 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
2020/03/29 10:49:16 [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration
Contributor guide
Assessment
This issue has not been assessed yet.