cockroachdb / cockroachdb/cockroach
gc: lack of observability into why GC jobs fail/don't run for system tables
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
_Created from CRDB-39149_.
Currently there is no observability into why GC jobs fail to run or do not run at all for system tables. Take as an example the table system.lease. This table is known to cause issues [due to this bug](https://github.com/cockroachdb/cockroach/pull/119466), where a pts is on the lease table (preventing the gc from working).
There are no log entries, no jobs created for such gc job and we have no way of knowing if the gc job failed due to pts. [The fix](https://github.com/cockroachdb/cockroach/pull/119466) is in 23.1.17, but the workaround is to lower the gc.ttl on the system.lease table. Unfortunately, there is no visibility into when these jobs fail or don’t run, as even after lowering the gc.ttl as recommended in the fix does not resolve this problem, if the gc job doesn’t actually remove the data.
Jira issue: CRDB-41043
Contributor guide
Assessment
This issue has not been assessed yet.