hashicorp / hashicorp/consul

Consul memory increasing and raft db increased upto 300G

Open
#9,076 0 comments 0 reactions 0 assignees View on GitHub
theme/consul-vault theme/performance
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

Consul memory gradually increasing to 25G and still remains on the same and raft DB increased up to 300G since the login tokens are not getting cleaned up even after 5hrs is the RCA that we have identified.

ttl set as 5hrs based on our application use-case. We have around 4 Million stale session tokens.

#### Reproduction Steps

Steps to reproduce this issue:

1. Run `vault login....` continuously to create 40L login tokens.

### Consul info for both Client and Server

agent:
check_monitors = 0
check_ttls = 1
checks = 1
services = 1
build:
prerelease =
revision = 39f93f01
version = 1.2.1
consul:
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 4
goroutines = 51
max_procs = 4
os = linux
version = go1.10.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 4
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 13
members = 6
query_queue = 0
query_time = 1

### Operating system and Environment details

CentOS Linux release 7.8.2003 (Core)
3.10.0-1127.8.2.el7.x86_64

Contributor guide

Open the contributing guide

Research direction

Reproduce the report by running `vault login....` continuously with a five-hour TTL and monitoring token cleanup, Consul memory, and raft DB size. Done means stale login tokens are removed after the TTL and the raft DB no longer grows indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.