etcd-io / etcd-io/etcd

Add a metric for the size of current live KV data

Open
#22,079 3 comments 0 reactions 0 assignees View on GitHub
type/feature
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
3d 1h
Merged PRs (30d)
44

Description

### What would you like to be added?

Add a Prometheus gauge that reports the logical size of the current live KV data, for example:

`etcd_mvcc_current_kv_size_in_bytes`

The value could be defined as the sum of `len(key) + len(value)` for the latest non-deleted version of every key. It should exclude historical revisions, tombstones, database fragmentation, and other backend storage overhead.

This would complement the existing metrics:

- `etcd_mvcc_db_total_size_in_use_in_bytes`, which includes live data, retained MVCC history, and backend overhead;
- `etcd_mvcc_db_total_size_in_bytes`, which reports physically allocated database size.

The metric should not introduce labels and would represent the local member's applied state. If maintaining an exact value would add significant startup, write, range-delete, or snapshot-restore overhead, an approximate value or an `etcd_debugging` metric would also be useful. I would appreciate guidance on which trade-off fits etcd's metrics policy.

### Why is this needed?

We operate an etcd cluster whose workload is dominated by overwrites. It has two main write patterns: frequent overwrites of small keys and less frequent overwrites of large keys. Auto-compaction uses a fixed revision-retention window.

While the frequent small-key writes were present, most revisions in that window were small and database usage remained stable. After those writes stopped, the overall write rate decreased, but large-key revisions made up a greater share of the retained history. `etcd_mvcc_db_total_size_in_use_in_bytes` therefore continued to grow.

The existing metrics showed the growth, but could not distinguish growth in current application data from accumulated MVCC history. We had to analyze a snapshot to identify the cause.

A current-live-data metric would make this distinction directly observable and help operators evaluate whether their auto-compaction retention policy matches their workload. The existing `etcd_debugging_mvcc_total_put_size_in_bytes` does not provide this view because it accumulates all put payloads rather than tracking the latest value of each key.

Contributor guide

Open the contributing guide

Research direction

Start by reading the implementations of the existing etcd_mvcc_db_total_size_in_use_in_bytes and etcd_mvcc_db_total_size_in_bytes metrics, then trace MVCC writes, range deletes, and snapshot restore handling. Define how the gauge accounts for the latest non-deleted key and value without labels, and verify that the chosen exact or approximate approach distinguishes live data from history and backend overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
databases, distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.