hashicorp / hashicorp/consul

Consul Server Pods Instability

Open
#22,531 2 comments 9 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

We are experiencing several persistent and critical stability issues when running Consul in large-scale production environments hosted in Azure Kubernetes Service (AKS). These issues do not reproduce in simple internal test environments, suggesting they are related to scale, network latency, or cloud-specific constraints.

Key problems include:

Consul raft.db corruption — one of the followers frequently falls behind and eventually gets out of sync or fails completely.

Random leader election failures — the leader is sometimes lost unexpectedly, even when all nodes appear healthy.

Backup intermittently fails — especially under cluster stress or heavy raft activity.

Consul agents do not start with priority — on node restarts or scaling events, Consul daemons often come up after other services, causing dependency failures.

We believe these problems stem from default configuration values that are not well-suited for high-scale cloud environments (e.g., AKS with many nodes and pods). Additional tuning and prioritization of Consul startup and raft reliability is needed.

---

#### Reproduction Steps

### Consul info for both Client and Server

[ERROR] agent.server.raft: failed to get log: error="log not found"

### Operating system and Environment details

Consul version: 1.17.3
Kubernetes version: 1.31
### Log Fragments

Contributor guide

Open the contributing guide

Research direction

Start by separating the reported raft.db corruption, leader-election failures, backup failures, and startup ordering issues using the Consul 1.17.3 and Kubernetes 1.31 details. Inspect the agent.server.raft error and collect TRACE logs, snapshot or backup-container logs, and AKS reproduction details under network delay or node restart. Done requires a reproducible failure with evidence identifying which behavior needs correction.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go, kubernetes
Domain
cloud, devops, distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.