cockroachdb / cockroachdb/cockroach
*,kv,storage: coalesced background scans
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
There are many various background tasks that require scanning a KV range, eg:
- Row-level TTL jobs
- MVCC GC
- Backups
- SQL stats collection
- Constructing a replication snapshot
- KV consistency checks
Additionally, storage engine compactions scan a more constrained scope (observing only keys from the input LSM levels) and don't respect KV range boundaries.
All these various background jobs are controlled by their own heuristics, schedules, etc. There may be opportunity to reduce the background CPU and read I/O costs by 'coalescing' these scans in some instances. A single `*pebble.Iterator` scanning across the keyspace can be used by multiple jobs that can all benefit from the scan.
There are many potential obstacles here:
- some operations have particular synchronization requirements that would make it difficult for them to share a LSM readState
- some operations are less tolerant to pacing (eg, replication)
- some operations buffer writes, and these operations are batched according to these buffers. other operations may not be able to be batched whatsoever
Jira issue: CRDB-30141
Contributor guide
Research direction
No files, tests, or entry points are named. Start by mapping the listed background jobs, storage-engine compactions, and their Pebble iterator usage, then investigate synchronization, pacing, and batching constraints. Done would require a defined coalescing design plus evidence that it reduces background CPU and read I/O without violating those constraints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100