cockroachdb / cockroachdb/cockroach

kv: index backfill never completes with ReplAC apply_to_all

Open
#135,343 1 comment 0 reactions 0 assignees View on GitHub
A-replication-admission-control-v2 branch-master C-bug O-perturbation T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

While running the backfill test, the backfill hung and nodes became stuck waiting for a snapshot.

**To Reproduce**

Run the following test:

```
PERTURBATION_OVERRIDE=acMode=fullBoth roachtest run perturbation/full/backfill
```

Additionally this test reproduces the issue as well:
https://github.com/cockroachdb/cockroach/pull/135339

**Additional data / screenshots**

The error in the logs is:
```
E241115 22:00:40.609821 23415709 kv/kvserver/queue.go:1198 ⋮ [T1,Vsystem,n3,raftsnapshot,s6,r7801/4:‹/Table/109/1/-781{9715…-7870…}›] 535505 error sending couldn't accept ‹range_id:7801 coordinator_replica: recipient_replica: delegated_sender: term:7 first_index:11993 sender_queue_name:RAFT_SNAPSHOT_QUEUE descriptor_generation:95 queue_on_delegate_len:-1 snap_id:9e4c2549-8a9c-4d99-8d92-99594f668bd8 ›: (n12,s24):1: remote couldn't accept snapshot 9e4c2549 at applied index 11993: ‹snapshot intersects existing range; initiated GC:› [n12,s24,r7924/4:‹/Table/109/1/-78{2340…-1418…}›] (incoming ‹/Table/109/1/-781{9715178531312532-7870688572937416}›)
```

This repeats at a high rate (~100/s)

[Cluster link](https://grafana.testeng.crdb.io/d/J-yAVzkVddd/detailed?orgId=1&refresh=5s&var-cluster=baptistcockroachlabscom-1731701126-01-n13cpu16sm&var-DS_PROMETHEUS=v9Zz2K6nz&var-node=All&var-rate_interval=30s&var-token_type=All&from=now-30m&to=now)

Jira issue: CRDB-44457

Epic CRDB-42900

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.