Add a metric for the size of current live KV data
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 44
Description
### What would you like to be added?
Add a Prometheus gauge that reports the logical size of the current live KV data, for example:
`etcd_mvcc_current_kv_size_in_bytes`
The value could be defined as the sum of `len(key) + len(value)` for the latest non-deleted version of every key. It should exclude historical revisions, tombstones, database fragmentation, and other backend storage overhead.
This would complement the existing metrics:
- `etcd_mvcc_db_total_size_in_use_in_bytes`, which includes live data, retained MVCC history, and backend overhead;
- `etcd_mvcc_db_total_size_in_bytes`, which reports physically allocated database size.
The metric should not introduce labels and would represent the local member's applied state. If maintaining an exact value would add significant startup, write, range-delete, or snapshot-restore overhead, an approximate value or an `etcd_debugging` metric would also be useful. I would appreciate guidance on which trade-off fits etcd's metrics policy.
### Why is this needed?
We operate an etcd cluster whose workload is dominated by overwrites. It has two main write patterns: frequent overwrites of small keys and less frequent overwrites of large keys. Auto-compaction uses a fixed revision-retention window.
While the frequent small-key writes were present, most revisions in that window were small and database usage remained stable. After those writes stopped, the overall write rate decreased, but large-key revisions made up a greater share of the retained history. `etcd_mvcc_db_total_size_in_use_in_bytes` therefore continued to grow.
The existing metrics showed the growth, but could not distinguish growth in current application data from accumulated MVCC history. We had to analyze a snapshot to identify the cause.
A current-live-data metric would make this distinction directly observable and help operators evaluate whether their auto-compaction retention policy matches their workload. The existing `etcd_debugging_mvcc_total_put_size_in_bytes` does not provide this view because it accumulates all put payloads rather than tracking the latest value of each key.
Contributor guide
Research direction
Start by reading the implementations of the existing etcd_mvcc_db_total_size_in_use_in_bytes and etcd_mvcc_db_total_size_in_bytes metrics, then trace MVCC writes, range deletes, and snapshot restore handling. Define how the gauge accounts for the latest non-deleted key and value without labels, and verify that the chosen exact or approximate approach distinguishes live data from history and backend overhead.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- databases, distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100