cockroachdb / cockroachdb/cockroach

kvserver: MVCC GC not happening after expected TTL after DROP

Open
#136,101 3 comments 0 reactions 0 assignees View on GitHub
branch-release-23.2 C-bug O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

During the course of a customer investigation on a v23.2.16 cluster, we were alerted to a `SCHEMA CHANGE GC` job that had been waiting since November 19th (6 days) for no obvious reason. We followed internal runbooks and determined that the reason the job hadn't run is because MVCC GC hadn't cleaned up yet, and it wasn't due to run for a very long time:

```
Checking garbage due...
r103248: 1 range keys, 0% garbage, GC in 3568.64 days
r103862: 2 range keys, 14.21% garbage, GC in 0.98 days
r104119: 1 range keys, 0.13% garbage, GC in 128.28 days
r6804: 4 range keys, 17.33% garbage, GC in 0.93 days
r80914: 1 range keys, 100% garbage, GC in 0.13 days
r8330: 4 range keys, 1.32% garbage, GC in 12.55 days
```

After manually running these through the MVCC GC queue, the job completed. This was previously thought to have been fixed in 23.2+, and only present in 23.1, or for 23.2 schema changes that were run in 23.1. This is not the case here - the cluster has been in 23.2 since May.

It seems that the range _should_ have a GC hint enabled after the schema change is done, but we did not find any:

```
"gc_hint": {
"latest_range_delete_timestamp": {},
"gc_timestamp": {},
"gc_timestamp_next": {}
}
```

A redacted copy of the range information that was waiting for 10 years is attached.

This is the job in question:

```
> show job 1022305863164592136;
job_id | job_type | description | statement | user_name | status | running_status | created | started | finished | modified | fraction_completed | error | coordinator_id | trace_id | last_run | next_run | num_runs | execution_errors
----------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------+-----------+----------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+--------------------+-------+----------------+---------------------+-------------------------------+-------------------------------+----------+-------------------
1022305863164592136 | SCHEMA CHANGE GC | GC for CREATE INDEX ON .public. ("" ASC, ASC, "" DESC) | | node | succeeded | NULL | 2024-11-19 21:56:30.463176+00 | 2024-11-19 21:56:30.643438+00 | 2024-11-25 11:26:31.941345+00 | 2024-11-25 11:26:31.941352+00 | 1 | | 3 | 6061693584673925224 | 2024-11-25 11:21:31.856972+00 | 2024-11-25 11:22:01.856972+00 | 1 | NULL
```

**To Reproduce**

This has not been reproduced, this has been observed in a customer environment.

**Expected behavior**

The expectation is that this cluster would no longer have been able to run into this behavior.

**Additional data / screenshots**
[r103248.json](https://github.com/user-attachments/files/17902599/r103248.json)

**Environment:**

Dedicated Advanced - v23.2.16

**Additional context**

Impact - this job would have stuck around for 10 years without manual intervention. This [runbook](https://cockroachlabs.atlassian.net/wiki/x/fY-IEgE) describes the intervention.

Jira issue: CRDB-44890

Epic CRDB-43310

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.