cockroachdb / cockroachdb/cockroach

sql/inspect: support periodic execution of INSPECT for ongoing corruption monitoring

Open
#172,003 2 comments 0 reactions 0 assignees View on GitHub
A-sql-scrub C-enhancement O-agent O-postmortem P-3 T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

Today there is no built-in way to run INSPECT periodically against a production
cluster on a recurring schedule. Operators who want to proactively monitor for
data corruption (e.g. index inconsistencies, uniqueness violations) have to
build their own scheduling on top of the INSPECT statement — managing what to
check, how often, when to run it during low-traffic windows, how to throttle
concurrency, and how to surface results.

This makes ongoing corruption monitoring difficult to adopt in production, even
though INSPECT is the mechanism we want users to rely on.

**Describe the solution you'd like**

A first-class way to run INSPECT periodically in a production cluster, with at
least the following characteristics:

- Schedulable (e.g. via the scheduled jobs subsystem) so the cluster runs
INSPECT automatically on a recurring cadence without external orchestration.
- Configurable scope (per-database, per-table, per-index) and cadence, with
reasonable defaults.
- Respects existing INSPECT controls (concurrency limits — see #165715,
protected timestamp behavior — see #171467, admission control) so periodic
runs don't destabilize the cluster.
- Results that are easy for operators and tooling to consume: persisted
findings, surfaced via metrics, and discoverable via `SHOW INSPECT ERRORS`.
- Sensible behavior when a scheduled run overlaps with a still-running
previous run.

**Describe alternatives you've considered**

- Documenting a recipe that uses external schedulers (cron, k8s CronJob, etc.)
to issue INSPECT statements. This works but pushes complexity onto every
operator and doesn't give us a consistent surface for results, metrics, or
throttling.
- Using the existing scheduled jobs subsystem (as we do for scheduled backups
and scheduled changefeeds) as the implementation vehicle — this is likely
the right starting point.

**Additional context**

This is intended as scaffolding to make INSPECT useful as an *ongoing*
corruption-monitoring tool, not just an on-demand operator command. Several
adjacent issues are relevant for capacity/safety of periodic runs:

- #165715 — concurrency control for simultaneous INSPECT executions
- #171467 — per-job protected timestamp scaling
- #160989 — row-counting on multiple tables/indexes per job

Epic CRDB-58926

Jira issue: CRDB-65138

Contributor guide

Open the contributing guide

Research direction

Start with the existing INSPECT statement and scheduled jobs subsystem, then review the related issues #165715, #171467, and #160989 for concurrency, protected timestamps, and multi-table scope. Done means a production-safe recurring INSPECT mechanism with configurable scope and cadence, overlap handling, and results available through persisted findings, metrics, and SHOW INSPECT ERRORS.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.