cockroachdb / cockroachdb/cockroach
kv: index backfill never completes with ReplAC apply_to_all
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
While running the backfill test, the backfill hung and nodes became stuck waiting for a snapshot.
**To Reproduce**
Run the following test:
```
PERTURBATION_OVERRIDE=acMode=fullBoth roachtest run perturbation/full/backfill
```
Additionally this test reproduces the issue as well:
https://github.com/cockroachdb/cockroach/pull/135339
**Additional data / screenshots**
The error in the logs is:
```
E241115 22:00:40.609821 23415709 kv/kvserver/queue.go:1198 ⋮ [T1,Vsystem,n3,raftsnapshot,s6,r7801/4:‹/Table/109/1/-781{9715…-7870…}›] 535505 error sending couldn't accept ‹range_id:7801 coordinator_replica: recipient_replica: delegated_sender: term:7 first_index:11993 sender_queue_name:RAFT_SNAPSHOT_QUEUE descriptor_generation:95 queue_on_delegate_len:-1 snap_id:9e4c2549-8a9c-4d99-8d92-99594f668bd8 ›: (n12,s24):1: remote couldn't accept snapshot 9e4c2549 at applied index 11993: ‹snapshot intersects existing range; initiated GC:› [n12,s24,r7924/4:‹/Table/109/1/-78{2340…-1418…}›] (incoming ‹/Table/109/1/-781{9715178531312532-7870688572937416}›)
```
This repeats at a high rate (~100/s)
[Cluster link](https://grafana.testeng.crdb.io/d/J-yAVzkVddd/detailed?orgId=1&refresh=5s&var-cluster=baptistcockroachlabscom-1731701126-01-n13cpu16sm&var-DS_PROMETHEUS=v9Zz2K6nz&var-node=All&var-rate_interval=30s&var-token_type=All&from=now-30m&to=now)
Jira issue: CRDB-44457
Epic CRDB-42900
Contributor guide
Assessment
This issue has not been assessed yet.