cockroachdb / cockroachdb/cockroach
kvserver: redundant MVCC GC runs in production systems
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We log whenever MVCC GC runs and finds that immediately after, the range is still eligible for GC:
https://github.com/cockroachdb/cockroach/blob/0ff29cc5f20b43faf3b58d5bbe83c3f329eb243f/pkg/kv/kvserver/mvcc_gc_queue.go#L783-L798
We also queue a stats recomputation in that case (which scans the entire range).
An analysis of the last week shows that we see this message around 2.5 times per second in our fleet, and that this predominantly happens on a few clusters.
Unfortunately, the log message does not indicate the reason for which the original MVCC GC run was carried out, which could help us understand if we're hitting an edge case such as this one
https://github.com/cockroachdb/cockroach/blob/765504b992e749e80a83a5e6a3d5ba6c556c12dd/pkg/kv/kvserver/mvcc_gc_queue.go#L526-L533
or others that exist in the code.
Either way, there is something we can improve here - extraneous MVCC GC runs are wasteful, especially when they come with a stats recomputation as well.
"Last seven days" analysis on [DD](https://us5.datadoghq.com/logs?query=service%3Acockroachdb%20-%40channel%3A%28STORAGE%20OR%20SESSIONS%20OR%20TELEMETRY%29%20%22GC%20still%20needed%20following%20GC%22&agg_m=count&agg_m_source=base&agg_q=%40friendly_id&agg_q_source=base&agg_t=count&cols=host%2Cservice&messageDisplay=inline&refresh_mode=sliding&storage=flex_tier&stream_sort=desc&top_n=50&top_o=top&viz=sunburst&x_missing=true&from_ts=1757409923534&to_ts=1758014723534&live=true)
Jira issue: CRDB-54496
Contributor guide
Assessment
This issue has not been assessed yet.