cockroachdb / cockroachdb/cockroach

*,kv,storage: coalesced background scans

Open
#107,643 0 comments 0 reactions 0 assignees View on GitHub
A-sql-table-stats A-storage C-enhancement T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

There are many various background tasks that require scanning a KV range, eg:
- Row-level TTL jobs
- MVCC GC
- Backups
- SQL stats collection
- Constructing a replication snapshot
- KV consistency checks

Additionally, storage engine compactions scan a more constrained scope (observing only keys from the input LSM levels) and don't respect KV range boundaries.

All these various background jobs are controlled by their own heuristics, schedules, etc. There may be opportunity to reduce the background CPU and read I/O costs by 'coalescing' these scans in some instances. A single `*pebble.Iterator` scanning across the keyspace can be used by multiple jobs that can all benefit from the scan.

There are many potential obstacles here:
- some operations have particular synchronization requirements that would make it difficult for them to share a LSM readState
- some operations are less tolerant to pacing (eg, replication)
- some operations buffer writes, and these operations are batched according to these buffers. other operations may not be able to be batched whatsoever

Jira issue: CRDB-30141

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by mapping the listed background jobs, storage-engine compactions, and their Pebble iterator usage, then investigate synchronization, pacing, and batching constraints. Done would require a defined coalescing design plus evidence that it reduces background CPU and read I/O without violating those constraints.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.