cockroachdb / cockroachdb/cockroach

kvserver: replicaGC'able replica causing stuck rangefeed

Open
#129,928 1 comment 0 reactions 0 assignees View on GitHub
A-kv-closed-timestamps A-kv-rangefeed branch-master C-bug P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

on a DRT test cluster, a lagging logical data replication process was observed. Using `crdb_internal.active_range_feeds`, this was narrowed down to a partial rangefeed's resolved timestamp lagging by 3h+, pointing at a particular replica on a particular node. In the range status page for that range, we saw that that range was in raft state Follower, and was quiescent. It was a removed learner and the moment we manually enqueued for replicaGC, the replica disappeared and the LDR lag resolved.

We grabbed a goroutine dump before enqueuing, and there is a side-eye snapshot as well as a number of screenshots at https://cockroachlabs.slack.com/archives/C0KB9Q03D/p1724359072165569.

**To Reproduce**

via @msbutler:

> We have only observed this on master, on a 40 node cluster. Once LDR is set up, it’s quite easy to repo by killing a node. I imagine you don’t need to set up LDR to repro this.
It’s also interesting that we haven’t seen this failure on roachtests that kill nodes, but have smaller deployments

**Expected behavior**

Replicas are replicaGC'ed within a few seconds of becoming eligible.

Maybe there is another angle here? But due to quiescence, the rangefeed on the replica itself couldn't be much more proactive, after all it doesn't know it's expendable.

If we remove quiescence, we have more options, but then replicaGC is also very proactive almost all of the time.

**Additional data / screenshots**

https://cockroachlabs.slack.com/archives/C0KB9Q03D/p1724359072165569

**Environment:**

**Additional context**

Jira issue: CRDB-41777

Contributor guide

Open the contributing guide

Research direction

Start with the kvserver replicaGC path and the crdb_internal.active_range_feeds output described in the report. Reproduce the behavior by killing a node in a large cluster, then inspect the range status for a quiescent removed learner. Done means eligible replicas are garbage-collected within seconds and the rangefeed lag resolves.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.