cockroachdb / cockroachdb/cockroach
kvserver: MVCC GC not happening after expected TTL after DROP
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
During the course of a customer investigation on a v23.2.16 cluster, we were alerted to a `SCHEMA CHANGE GC` job that had been waiting since November 19th (6 days) for no obvious reason. We followed internal runbooks and determined that the reason the job hadn't run is because MVCC GC hadn't cleaned up yet, and it wasn't due to run for a very long time:
```
Checking garbage due...
r103248: 1 range keys, 0% garbage, GC in 3568.64 days
r103862: 2 range keys, 14.21% garbage, GC in 0.98 days
r104119: 1 range keys, 0.13% garbage, GC in 128.28 days
r6804: 4 range keys, 17.33% garbage, GC in 0.93 days
r80914: 1 range keys, 100% garbage, GC in 0.13 days
r8330: 4 range keys, 1.32% garbage, GC in 12.55 days
```
After manually running these through the MVCC GC queue, the job completed. This was previously thought to have been fixed in 23.2+, and only present in 23.1, or for 23.2 schema changes that were run in 23.1. This is not the case here - the cluster has been in 23.2 since May.
It seems that the range _should_ have a GC hint enabled after the schema change is done, but we did not find any:
```
"gc_hint": {
"latest_range_delete_timestamp": {},
"gc_timestamp": {},
"gc_timestamp_next": {}
}
```
A redacted copy of the range information that was waiting for 10 years is attached.
This is the job in question:
```
> show job 1022305863164592136;
job_id | job_type | description | statement | user_name | status | running_status | created | started | finished | modified | fraction_completed | error | coordinator_id | trace_id | last_run | next_run | num_runs | execution_errors
----------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------+-----------+----------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+--------------------+-------+----------------+---------------------+-------------------------------+-------------------------------+----------+-------------------
1022305863164592136 | SCHEMA CHANGE GC | GC for CREATE INDEX ON .public. ("" ASC, ASC, "" DESC) | | node | succeeded | NULL | 2024-11-19 21:56:30.463176+00 | 2024-11-19 21:56:30.643438+00 | 2024-11-25 11:26:31.941345+00 | 2024-11-25 11:26:31.941352+00 | 1 | | 3 | 6061693584673925224 | 2024-11-25 11:21:31.856972+00 | 2024-11-25 11:22:01.856972+00 | 1 | NULL
```
**To Reproduce**
This has not been reproduced, this has been observed in a customer environment.
**Expected behavior**
The expectation is that this cluster would no longer have been able to run into this behavior.
**Additional data / screenshots**
[r103248.json](https://github.com/user-attachments/files/17902599/r103248.json)
**Environment:**
Dedicated Advanced - v23.2.16
**Additional context**
Impact - this job would have stuck around for 10 years without manual intervention. This [runbook](https://cockroachlabs.atlassian.net/wiki/x/fY-IEgE) describes the intervention.
Jira issue: CRDB-44890
Epic CRDB-43310
Contributor guide
Assessment
This issue has not been assessed yet.