Consul memory increasing and raft db increased upto 300G
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
Consul memory gradually increasing to 25G and still remains on the same and raft DB increased up to 300G since the login tokens are not getting cleaned up even after 5hrs is the RCA that we have identified.
ttl set as 5hrs based on our application use-case. We have around 4 Million stale session tokens.
#### Reproduction Steps
Steps to reproduce this issue:
1. Run `vault login....` continuously to create 40L login tokens.
### Consul info for both Client and Server
agent:
check_monitors = 0
check_ttls = 1
checks = 1
services = 1
build:
prerelease =
revision = 39f93f01
version = 1.2.1
consul:
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 4
goroutines = 51
max_procs = 4
os = linux
version = go1.10.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 4
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 13
members = 6
query_queue = 0
query_time = 1
### Operating system and Environment details
CentOS Linux release 7.8.2003 (Core)
3.10.0-1127.8.2.el7.x86_64
Contributor guide
Research direction
Reproduce the report by running `vault login....` continuously with a five-hour TTL and monitoring token cleanup, Consul memory, and raft DB size. Done means stale login tokens are removed after the TTL and the raft DB no longer grows indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100