cockroachdb / cockroachdb/cockroach
kvserver: replicaGC'able replica causing stuck rangefeed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
on a DRT test cluster, a lagging logical data replication process was observed. Using `crdb_internal.active_range_feeds`, this was narrowed down to a partial rangefeed's resolved timestamp lagging by 3h+, pointing at a particular replica on a particular node. In the range status page for that range, we saw that that range was in raft state Follower, and was quiescent. It was a removed learner and the moment we manually enqueued for replicaGC, the replica disappeared and the LDR lag resolved.
We grabbed a goroutine dump before enqueuing, and there is a side-eye snapshot as well as a number of screenshots at https://cockroachlabs.slack.com/archives/C0KB9Q03D/p1724359072165569.
**To Reproduce**
via @msbutler:
> We have only observed this on master, on a 40 node cluster. Once LDR is set up, it’s quite easy to repo by killing a node. I imagine you don’t need to set up LDR to repro this.
It’s also interesting that we haven’t seen this failure on roachtests that kill nodes, but have smaller deployments
**Expected behavior**
Replicas are replicaGC'ed within a few seconds of becoming eligible.
Maybe there is another angle here? But due to quiescence, the rangefeed on the replica itself couldn't be much more proactive, after all it doesn't know it's expendable.
If we remove quiescence, we have more options, but then replicaGC is also very proactive almost all of the time.
**Additional data / screenshots**
https://cockroachlabs.slack.com/archives/C0KB9Q03D/p1724359072165569
**Environment:**
**Additional context**
Jira issue: CRDB-41777
Contributor guide
Research direction
Start with the kvserver replicaGC path and the crdb_internal.active_range_feeds output described in the report. Reproduce the behavior by killing a node in a large cluster, then inspect the range status for a quiescent removed learner. Done means eligible replicas are garbage-collected within seconds and the rangefeed lag resolves.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100