Service can’t register in consul after unknown failure in kubernetes: Error while renaming Node ID
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
Greetings,
#### Overview of the Issue
We are running our service in kubernetes pod with sidecard containing consul agent, also main consul cluster as statefulset.
When any casual problem occurs on service side - k8s reloading pods and service registers in consul without problems.
But we faced some floating problem - something happens (maybe network issues?), then service can't reregister in consul and got errors:
`rpc error making call: failed inserting node: Error while renaming Node ID: "4833aa15-8428-1d1a-46d8-9dba157dbc60": Node name xxx.xxx.xxx is reserved by node c5dc5b48-f105-79f0-7910-3de6629fddd0 with name xxx.xxx.xx`
Also service returning 200 - OK healthcheck, but consul forbids registering.
### Service, consul info for both Client and Server
Service:
* Spring Boot - 2.2.4
* Spring Cloud - Hoxton
Sidecar consul agent cli:
```
- consul
- agent
-retry-join={{.Values.global.consul.server}}
-datacenter={{ template "alv.consul.datacenter" . }}
-data-dir=/consul/data
```
Also '{"leave_on_terminate": true}'
Client info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 1
services = 1
build:
prerelease =
revision = a417fe51
version = 1.9.0
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 16
goroutines = 64
max_procs = 16
os = linux
version = go1.15.5
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 14307
members = 25
query_queue = 0
query_time = 1
```
Server info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = a417fe51
version = 1.9.0
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 172.20.77.167:8300
server = true
raft:
applied_index = 1441965
commit_index = 1441965
fsm_pending = 0
last_contact = 0
last_log_index = 1441965
last_log_term = 4
last_snapshot_index = 1428703
last_snapshot_term = 4
latest_configuration = [{Suffrage:Voter ID:539947fd-7fae-e0f8-6358-4ffc1d6baf35 Address:172.20.77.167:8300} {Suffrage:Voter ID:b0b7bda8-ff0d-63d6-a0d3-a75409d66b83 Address:172.20.72.68:8300} {Suffrage:Voter ID:8c7e059f-dd66-58ae-6c8d-e5fd28288867 Address:172.20.73.189:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 4
runtime:
arch = amd64
cpu_count = 24
goroutines = 462
max_procs = 24
os = linux
version = go1.15.5
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 15
health_score = 0
intent_queue = 0
left = 4
member_time = 14298
members = 44
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 8
members = 3
query_queue = 0
query_time = 1
```
### Spring Boot Config
```
spring:
application:
name: ActivationLauncherValidator
cloud:
consul:
host: ${consul.host}
port: ${consul.port}
discovery:
prefer-ip-address: true
heartbeat:
enabled: false
instance-id: ${spring.application.name}-${spring.cloud.client.hostname}-${server.port}
health-check-critical-timeout: 5000m
health-check-interval: 30s
health-check-timeout: 5s
health-check-path: /health
tags: apiv1
```
### Operating system and Environment details
Service is running on buster-slim



Contributor guide
Research direction
No repository file, test, or code entry point is identified. Start by reproducing the registration failure with the reported Consul 1.9.0 client/server setup and Kubernetes sidecar configuration, then determine the lifecycle condition behind the reserved Node ID error; done means documenting a confirmed cause and actionable resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, linux, spring-boot
- Domain
- cloud, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100