Consul Server Pods Instability
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
We are experiencing several persistent and critical stability issues when running Consul in large-scale production environments hosted in Azure Kubernetes Service (AKS). These issues do not reproduce in simple internal test environments, suggesting they are related to scale, network latency, or cloud-specific constraints.
Key problems include:
Consul raft.db corruption — one of the followers frequently falls behind and eventually gets out of sync or fails completely.
Random leader election failures — the leader is sometimes lost unexpectedly, even when all nodes appear healthy.
Backup intermittently fails — especially under cluster stress or heavy raft activity.
Consul agents do not start with priority — on node restarts or scaling events, Consul daemons often come up after other services, causing dependency failures.
We believe these problems stem from default configuration values that are not well-suited for high-scale cloud environments (e.g., AKS with many nodes and pods). Additional tuning and prioritization of Consul startup and raft reliability is needed.
---
#### Reproduction Steps
### Consul info for both Client and Server
[ERROR] agent.server.raft: failed to get log: error="log not found"
### Operating system and Environment details
Consul version: 1.17.3
Kubernetes version: 1.31
### Log Fragments
Contributor guide
Research direction
Start by separating the reported raft.db corruption, leader-election failures, backup failures, and startup ordering issues using the Consul 1.17.3 and Kubernetes 1.31 details. Inspect the agent.server.raft error and collect TRACE logs, snapshot or backup-container logs, and AKS reproduction details under network delay or node restart. Done requires a reproducible failure with evidence identifying which behavior needs correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go, kubernetes
- Domain
- cloud, devops, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100