hashicorp / hashicorp/consul

Handle agent IP address changes on restart (no Raft quorum) in data recovery scenario

Open
#10,678 7 comments 0 reactions 0 assignees View on GitHub
theme/internals theme/operator-usability type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

Hello,

I have a consul cluster deployed in a kubernetes cluster.
When stopping the Consul cluster and starting it again, every pod has a new IP address, and the raft election fails:

[ERROR] agent.server.raft: failed to make requestVote RPC: target="{Voter b91......... 172.XX.XX.XX.:8300}" error="dial tcp ->72.XX.XX.XX.:8300: i/o timeout"
(172.XX.XX.XX is assign to no one)

The only thing I can do is to delete every volume and restore a snapshot file, and everything is good again after that. But it is obviously time consuming.

How can I tell consul that the IP address of each pod changes at every restart ? Is there anything I can write in my values helm chart to do that ?

Ty and have a nice day

Contributor guide

Open the contributing guide

Research direction

Start by examining the Consul Kubernetes deployment and the Helm values involved in restarting pods with changed IP addresses, then trace how the Raft peer addresses are persisted. Reproduce the restart and recovery scenario; done means the cluster regains quorum without deleting volumes or restoring a snapshot.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.