Leader election after IP change causes RPCs to be sent to stale IP addresses
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.2.6 (a6c6b475db5073e33885377b4a5c733e1161020c)
### Operating system and Environment details
Ubuntu 20.04.3 LTS (Focal)
5.4.0-109-generic #123-Ubuntu
4-machine cluster with client+server on each as well as a consul agent on each
### Issue
When the members of the cluster stop and restart with new IPs, they correctly connect back to each other via Consul (because the Consul nodes find each other with hostnames).
However, they always fail to elect a leader (even though `bootstrap_expect` is 4):
```
Name Address Port
jobhost.global 192.168.0.102 4648
jobhost2.global 192.168.0.150 4648
jobhost3.global 192.168.0.156 4648
jobhost4.global 192.168.0.47 4648
Error determining leaders: 1 error occurred:
* Region "global": Unexpected response code: 500 (No cluster leader)
```
Looking at the logs, this seems to be because votes are being sent to stale/incorrect IP addresses:
```
requestVote RPC: target="{Voter 192.168.0.23:4647 192.168.0.23:4647}" error="dial tcp 192.168.0.23:4647: connect: no route to host"
requestVote RPC: target="{Voter 192.168.0.100:4647 192.168.0.100:4647}" error="dial tcp 192.168.0.100:4647: connect: no route to host"
requestVote RPC: target="{Voter 192.168.0.23:4647 192.168.0.23:4647}" error="dial tcp 192.168.0.23:4647: connect: no route to host"
requestVote RPC: target="{Voter 192.168.0.100:4647 192.168.0.100:4647}" error="dial tcp 192.168.0.100:4647: connect: no route to host"
requestVote RPC: target="{Voter 192.168.0.101:4647 192.168.0.101:4647}" error="dial tcp 192.168.0.101:4647: connect: no route to host"
requestVote RPC: target="{Voter 192.168.0.101:4647 192.168.0.101:4647}" error="dial tcp 192.168.0.101:4647: connect: no route to host"
nomad.raft: Election timeout reached, restarting election
```
I've verified that this information is not coming from Consul, because there are no longer any services (e.g. 'registered by Nomad') associated with these IPs.
I can only assume these addresses come from the actual raft log?
The Nomad configs all look like this so there is no hardcoded retry join etc:
```
data_dir = "/opt/nomad/data"
bind_addr = "0.0.0.0"
datacenter = "redacted"
name = "redacted"
server {
enabled = true
bootstrap_expect = 4
job_gc_interval = "48h"
job_gc_threshold = "24h"
eval_gc_threshold = "48h"
deployment_gc_threshold = "24h"
}
client {
enabled = true
servers = ["127.0.0.1"]
gc_max_allocs = 150
# Host volumes omitted
}
plugin "docker" {
config {
allow_caps = ["net_raw", "net_admin", "audit_write", "chown", "dac_override", "fowner", "fsetid", "kill", "mknod", "net_bind_service", "setfcap", "setgid", "setpcap", "setuid", "sys_chroot"]
}
}
vault {
enabled = true
ca_path = "/etc/nomad.d/vault_tls/ca.crt"
cert_file = "/etc/nomad.d/vault_tls/tls.crt"
key_file = "/etc/nomad.d/vault_tls/tls.key"
}
```
### Reproduction steps
Set up a cluster where nodes find each other via Consul and then restart each machine changing its IP in the process
### Summary
This scenario has now happened multiple times and each time the only solution is `peers.json` with the updated IPs. Often it's the case I have to forcefully remove stale catalog information from Consul as well (e.g. where Nomad has exited in a non-graceful manner)
- Am I misunderstanding how the cluster should behave here? e.g. is it actually _expected_ to not be resilient with IP changes?
- If an outage _is_ the expected result, is there some simpler way than peers.json etc. to inform Nomad that the address has changed?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Nomad Raft requestVote RPC logs and reproduce the reported restart-with-new-IP scenario using Consul-based discovery. Trace where the stale voter addresses persist, then verify that the cluster elects a leader and no longer sends RPCs to the old IPs; the report mentions peers.json as the current workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100