cockroachdb / cockroachdb/cockroach

kvserver: redundant MVCC GC runs in production systems

Open
#153,564 0 comments 0 reactions 0 assignees View on GitHub
A-kv-transactions C-investigation P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We log whenever MVCC GC runs and finds that immediately after, the range is still eligible for GC:

https://github.com/cockroachdb/cockroach/blob/0ff29cc5f20b43faf3b58d5bbe83c3f329eb243f/pkg/kv/kvserver/mvcc_gc_queue.go#L783-L798

We also queue a stats recomputation in that case (which scans the entire range).

An analysis of the last week shows that we see this message around 2.5 times per second in our fleet, and that this predominantly happens on a few clusters.

Unfortunately, the log message does not indicate the reason for which the original MVCC GC run was carried out, which could help us understand if we're hitting an edge case such as this one

https://github.com/cockroachdb/cockroach/blob/765504b992e749e80a83a5e6a3d5ba6c556c12dd/pkg/kv/kvserver/mvcc_gc_queue.go#L526-L533

or others that exist in the code.

Either way, there is something we can improve here - extraneous MVCC GC runs are wasteful, especially when they come with a stats recomputation as well.

"Last seven days" analysis on [DD](https://us5.datadoghq.com/logs?query=service%3Acockroachdb%20-%40channel%3A%28STORAGE%20OR%20SESSIONS%20OR%20TELEMETRY%29%20%22GC%20still%20needed%20following%20GC%22&agg_m=count&agg_m_source=base&agg_q=%40friendly_id&agg_q_source=base&agg_t=count&cols=host%2Cservice&messageDisplay=inline&refresh_mode=sliding&storage=flex_tier&stream_sort=desc&top_n=50&top_o=top&viz=sunburst&x_missing=true&from_ts=1757409923534&to_ts=1758014723534&live=true)

Jira issue: CRDB-54496

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.