cockroachdb / cockroachdb/cockroach

kv: slow replication on 150 node cluster.

Open
#134,194 0 comments 0 reactions 0 assignees View on GitHub
branch-release-24.3 C-bug O-24.3-scale-testing O-testcluster P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

On the` drt-scale` 150-node cluster, we observed slow up-replication of underreplicated ranges. This issue occurred while the cluster was idle, with no import or foreground SQL traffic running on the cluster.

Error logs are full of following error: [datadog link](https://us5.datadoghq.com/logs?query=cluster%3Adrt-scale%20status%3Aerror%20-%22background%20error%3A%20pebble%3A%20compaction%20cancelled%20by%20a%20concurrent%20operation%2C%20will%20retry%20compaction%22%20%22failed%20to%20replicate%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&event=AgAAAZL2k7lsd15yywAAAAAAAAAYAAAAAEFaTDJrN2xzQUFEc0V1blpaWXVuUGdBUAAAACQAAAAAMDE5MmY2OTQtNjBjZC00MzBkLTljMDctYjkyNzIxMTIwM2I0&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=flex_tier&stream_sort=time%2Casc&viz=stream&from_ts=1730713726200&to_ts=1730714169626&live=false)
```
failed to replicate after 5 retries
operation "‹replicate› queue process replica 579849" timed out after 1m20.003s (given timeout 1m20s): failed to replicate after 5 retries
```

The process of upreplicating from 40K to 5K ranges took 30 minutes, and similarly, it took 30 minutes to reduce the number of underreplicated ranges from 5K to 2K.

Screenshot 2024-11-04 at 3 26 52 PM
Screenshot 2024-11-04 at 3 31 49 PM

Jira issue: CRDB-43974

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.