Handle agent IP address changes on restart (no Raft quorum) in data recovery scenario
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
Hello,
I have a consul cluster deployed in a kubernetes cluster.
When stopping the Consul cluster and starting it again, every pod has a new IP address, and the raft election fails:
[ERROR] agent.server.raft: failed to make requestVote RPC: target="{Voter b91......... 172.XX.XX.XX.:8300}" error="dial tcp ->72.XX.XX.XX.:8300: i/o timeout"
(172.XX.XX.XX is assign to no one)
The only thing I can do is to delete every volume and restore a snapshot file, and everything is good again after that. But it is obviously time consuming.
How can I tell consul that the IP address of each pod changes at every restart ? Is there anything I can write in my values helm chart to do that ?
Ty and have a nice day
Contributor guide
Research direction
Start by examining the Consul Kubernetes deployment and the Helm values involved in restarting pods with changed IP addresses, then trace how the Raft peer addresses are persisted. Reproduce the restart and recovery scenario; done means the cluster regains quorum without deleting volumes or restoring a snapshot.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100