cockroachdb / cockroachdb/cockroach

kvserver: replicas on decommissioning node never being replaced

Open
#130,199 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution branch-release-23.1 C-bug O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

We have observed replicas on a decommissioning node (<3) never getting replaced by the corresponding range leaseholder node, over 80 minutes.

The stall was resolved by manually enqueueing the blocking ranges into the leaseholder's replicate queue.

This is surprising, as the replica scanner should check each replica once every 10 minutes, against all store replica queues. By manually enqueueing the ranges via the advanced debug page, without skipping the should queue check, it demonstrates that if the scanner is calling `shouldQueue` on the replica, it would have been enqueued. There is no indication that this occurred.

**To Reproduce**

Attempts at reproducing the issue haven't been successful so far. The methods tested are shown below.

Details

Setup the cluster and run the workload:

```
export cluster=austen-decom-repro
roachprod create $cluster -n 41
roachprod put $cluster ./artifacts/cockroach cockroach
roachprod start $cluster:1-40
rp sql $cluster:1 -- -e 'CREATE DATABASE kv2'
rp sql $cluster:1 -- -e 'ALTER RANGE default CONFIGURE ZONE USING num_replicas = 5'
rp sql $cluster:1 -- -e 'ALTER DATABASE kv2 CONFIGURE ZONE USING num_replicas = 5'
rp run $cluster:1 -- './cockroach workload run kv --init --splits=6000 --min-block-bytes=16384 --max-block-bytes=16384 --insert-count=100000000 --max-rate=100 {pgurl:1}'
```

Chain decomissions:

```bash
for iteration in $(seq 1 10000); do
echo "Starting iteration $iteration"

# Loop through nodes
for node in $(seq 20 40); do
echo "Processing node $node (Iteration $iteration)"

# Run drain command
rp run $cluster:$node -- './cockroach node drain --insecure --self'

# Run decommission command
rp run $cluster:$node -- './cockroach node decommission --insecure --self'

# Wipe the node
rp wipe $cluster:$node

# Put artifacts
rp put $cluster:$node ./artifacts/cockroach

# Start the node
rp start $cluster:$node

echo "Finished processing node $node (Iteration $iteration)"
echo "------------------------"
done

echo "Finished iteration $iteration"
echo "========================"
done
```

Restart a node every 3 minutes:

```bash
for iteration in $(seq 1 10000); do
# Loop through nodes
for node in $(seq 2 19); do
echo "Restarting node $node (Iteration $iteration)"

# Restart the node
rp stop $cluster:$node
# Start the node
rp start $cluster:$node

sleep 180

echo "Finished restarting node $node (Iteration $iteration)"
echo "------------------------"
done

echo "Finished iteration $iteration"
echo "========================"
done
```


**Expected behavior**

The replica scanner enqueues decommissioning ranges into the replicate queue at minimum every `replica_count * 100ms` (min scanner interval) duration, or 10 minutes, whichever is greater.

Decommissioning doesn't stall due to this.

**Environment:**
- CockroachDB version `v23.1.22`, however it may affect other versions, it has only occurred on a `v23.1.22` cluster

**Additional context**

Manual intervention is required to complete a decommission.

Jira issue: CRDB-41920

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.