cockroachdb / cockroachdb/cockroach
webui: show protected timestamps in DB Console
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Protected timestamps are used by various subsystems to "lock" MVCC history and prevent it from being garbage collected. For example, incremental backups need this to be able to back up the changes since the previous backup, and changefeeds need this in order to stream the changes that have happened since a changefeed paused or failed.
However, we've often seen such records get left behind indefinitely, such that we didn't run garbage collection for several months, increasing disk usage and reducing scan performance.
We should improve our observability of these protected timestamps. A few ideas:
* Expose relevant time series metrics
* `spanconfig.kvsubscriber.oldest_protected_record_nanos`: the age of the oldest protected timestamp record, in nanoseconds
* `spanconfig.kvsubscriber.protected_record_count`: the number of protected timestamp records
* Show a listing of protected timestamp records, obtained from `system.protected_ts_records`.
* Show which protected timestamps jobs are holding onto (join job ID with decoded `system.protected_ts_records.meta`).
* Full list of PTS records and their keyspans in the job detail.
* Number/age/spans of protected timestamps in job listing.
I'll leave it to @irfansharif and cluster observability to flesh out the details here. CDC and DR also have additional information/metrics/data about these protected timestamps, coordinate with them as well.
Jira issue: CRDB-28224
Contributor guide
Assessment
This issue has not been assessed yet.