etcd-io / etcd-io/etcd

defrag stops the process completely for noticeable duration

Open
#9,222 23 comments 0 reactions 0 assignees View on GitHub
area/performance stage/tracked
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

# What I observed

The defrag command blocks all the requests handled by the process for a while when the process has much data and incoming requests.
New requests and ongoing streaming requests such as `Watch()` can be canceled by their deadlines and the cluster can be unstable.

# Side effects

This "stop the world" behavior makes the operation of etcd clusters harder. We have the 8GB limitation of the DB size and need to control the DB size not to reach the border. However, clusters won't release DB space even after compaction until you run the defrag command. That means you cannot track this important metric without virtually stopping every etcd processe periodically.

(Btw, can we add a new metric for "actual DB usage"?)

CC/ @xiang90

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the defrag command while the process handles incoming and streaming requests such as Watch(). Investigate how defragmentation affects request handling and whether actual database usage can be measured. Done should mean defragmentation no longer causes unacceptable request disruption, with any new usage metric defined and validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.