cockroachdb / cockroachdb/cockroach
kv: slow replication on 150 node cluster.
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
On the` drt-scale` 150-node cluster, we observed slow up-replication of underreplicated ranges. This issue occurred while the cluster was idle, with no import or foreground SQL traffic running on the cluster.
Error logs are full of following error: [datadog link](https://us5.datadoghq.com/logs?query=cluster%3Adrt-scale%20status%3Aerror%20-%22background%20error%3A%20pebble%3A%20compaction%20cancelled%20by%20a%20concurrent%20operation%2C%20will%20retry%20compaction%22%20%22failed%20to%20replicate%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&event=AgAAAZL2k7lsd15yywAAAAAAAAAYAAAAAEFaTDJrN2xzQUFEc0V1blpaWXVuUGdBUAAAACQAAAAAMDE5MmY2OTQtNjBjZC00MzBkLTljMDctYjkyNzIxMTIwM2I0&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=flex_tier&stream_sort=time%2Casc&viz=stream&from_ts=1730713726200&to_ts=1730714169626&live=false)
```
failed to replicate after 5 retries
operation "‹replicate› queue process replica 579849" timed out after 1m20.003s (given timeout 1m20s): failed to replicate after 5 retries
```
The process of upreplicating from 40K to 5K ranges took 30 minutes, and similarly, it took 30 minutes to reduce the number of underreplicated ranges from 5K to 2K.
Jira issue: CRDB-43974
Contributor guide
Assessment
This issue has not been assessed yet.