cockroachdb / cockroachdb/cockroach

gc: lack of observability into why GC jobs fail/don't run for system tables

Open
#128,514 0 comments 0 reactions 0 assignees View on GitHub
A-kv C-enhancement O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

_Created from CRDB-39149_.

Currently there is no observability into why GC jobs fail to run or do not run at all for system tables. Take as an example the table system.lease. This table is known to cause issues [due to this bug](https://github.com/cockroachdb/cockroach/pull/119466), where a pts is on the lease table (preventing the gc from working).

There are no log entries, no jobs created for such gc job and we have no way of knowing if the gc job failed due to pts. [The fix](https://github.com/cockroachdb/cockroach/pull/119466) is in 23.1.17, but the workaround is to lower the gc.ttl on the system.lease table. Unfortunately, there is no visibility into when these jobs fail or don’t run, as even after lowering the gc.ttl as recommended in the fix does not resolve this problem, if the gc job doesn’t actually remove the data.

Jira issue: CRDB-41043

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.