ACL token leakage: Consul agents fail to logout, leading to ~20k stale tokens and Raft performance impact
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
A large number of ACL tokens accumulate due to Consul agents not performing a proper logout.
This results in increased Raft commit latency and resource consumption.
On a customer cluster, ~20,000 stale tokens had accumulated. After manual cleanup, Raft commit latency was reduced, but not significantly.
The problem originates from Consul itself rather than the application layer.
---
#### Reproduction Steps
1. Run Consul agents integrated with ESO discovery.
2. Perform repeated login/auth flows.
3. Observe that tokens are not properly released/logged out.
4. Over time, thousands of stale tokens accumulate.
### Consul info for both Client and Server
There is a noticeable degradation in commit performance (1–4 seconds per commit), resulting in increased load on the Consul servers.
### Operating system and Environment details
Consul version: 1.17.3
Kubernetes version: 1.31
### Log Fragments
Contributor guide
Research direction
The payload names no source files, entry points, or tests. Start by reproducing the repeated ACL login/logout flow with Consul 1.17.3 and ESO discovery, then trace token accumulation and its effect on Raft commit latency. Done means agents release or invalidate tokens correctly and a regression check covers the lifecycle.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- authorization, distributed-systems, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100