cockroachdb / cockroachdb/cockroach

webui: show protected timestamps in DB Console

Open
#103,831 4 comments 0 reactions 0 assignees View on GitHub
A-webui C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Protected timestamps are used by various subsystems to "lock" MVCC history and prevent it from being garbage collected. For example, incremental backups need this to be able to back up the changes since the previous backup, and changefeeds need this in order to stream the changes that have happened since a changefeed paused or failed.

However, we've often seen such records get left behind indefinitely, such that we didn't run garbage collection for several months, increasing disk usage and reducing scan performance.

We should improve our observability of these protected timestamps. A few ideas:

* Expose relevant time series metrics
* `spanconfig.kvsubscriber.oldest_protected_record_nanos`: the age of the oldest protected timestamp record, in nanoseconds
* `spanconfig.kvsubscriber.protected_record_count`: the number of protected timestamp records
* Show a listing of protected timestamp records, obtained from `system.protected_ts_records`.
* Show which protected timestamps jobs are holding onto (join job ID with decoded `system.protected_ts_records.meta`).
* Full list of PTS records and their keyspans in the job detail.
* Number/age/spans of protected timestamps in job listing.

I'll leave it to @irfansharif and cluster observability to flesh out the details here. CDC and DR also have additional information/metrics/data about these protected timestamps, coordinate with them as well.

Jira issue: CRDB-28224

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.