cockroachdb / cockroachdb/cockroach

PCR: gradual reduction in throughput during long initial backfill

Open
#120,338 2 comments 0 reactions 0 assignees View on GitHub
C-bug O-testcluster P-2 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

A cluster replicating from a 35TB source was observed to gradually be ingesting at slower and slower rates over the first 8 hours of its initial backfill scan. At the 8h mark it was ingesting at an aggregate rate across all 9 nodes of approximately 35MB/s of SST bytes, or about 4MB/s/node, down from about 3x that initially.

Jira issue: CRDB-36642

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the reported 35TB replication scenario and the linked Jira issue CRDB-36642, focusing on the initial backfill scan and its declining SST ingestion rate. No source file or test is identified; progress would require locating the relevant backfill path, reproducing or explaining the throughput decline, and defining a verified improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.