cockroachdb / cockroachdb/cockroach
PCR: gradual reduction in throughput during long initial backfill
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
A cluster replicating from a 35TB source was observed to gradually be ingesting at slower and slower rates over the first 8 hours of its initial backfill scan. At the 8h mark it was ingesting at an aggregate rate across all 9 nodes of approximately 35MB/s of SST bytes, or about 4MB/s/node, down from about 3x that initially.
Jira issue: CRDB-36642
Contributor guide
Research direction
Start by reviewing the reported 35TB replication scenario and the linked Jira issue CRDB-36642, focusing on the initial backfill scan and its declining SST ingestion rate. No source file or test is identified; progress would require locating the relevant backfill path, reproducing or explaining the throughput decline, and defining a verified improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100