hashicorp / hashicorp/consul

ACL token leakage: Consul agents fail to logout, leading to ~20k stale tokens and Raft performance impact

Open
#22,613 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

A large number of ACL tokens accumulate due to Consul agents not performing a proper logout.
This results in increased Raft commit latency and resource consumption.

On a customer cluster, ~20,000 stale tokens had accumulated. After manual cleanup, Raft commit latency was reduced, but not significantly.

The problem originates from Consul itself rather than the application layer.

---

#### Reproduction Steps

1. Run Consul agents integrated with ESO discovery.
2. Perform repeated login/auth flows.
3. Observe that tokens are not properly released/logged out.
4. Over time, thousands of stale tokens accumulate.

### Consul info for both Client and Server

There is a noticeable degradation in commit performance (1–4 seconds per commit), resulting in increased load on the Consul servers.

### Operating system and Environment details

Consul version: 1.17.3
Kubernetes version: 1.31

### Log Fragments

Contributor guide

Open the contributing guide

Research direction

The payload names no source files, entry points, or tests. Start by reproducing the repeated ACL login/logout flow with Consul 1.17.3 and ESO discovery, then trace token accumulation and its effect on Raft commit latency. Done means agents release or invalidate tokens correctly and a regression check covers the lifecycle.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
authorization, distributed-systems, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.