cockroachdb / cockroachdb/cockroach
sql: automatically validate new index recommendations using partial visibility
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
CockroachDB produces index recommendations based on query workloads, but
acting on those recommendations is risky: creating a new index can change
query plans in unexpected ways. A recommended index that looks beneficial
in isolation may cause regressions for other statements (e.g. by being
chosen over a better existing index, or by adding write amplification that
isn't justified by read gains). Today, the operator bears all of the risk
when applying a recommendation, and there is no built-in mechanism to
roll back automatically if performance regresses.
**Describe the solution you'd like**
A feature that automatically tests an index recommendation in production
and decides whether to keep, drop, or fully enable it based on observed
performance. The proposed mechanism uses *partially visible indexes* as
the experimentation primitive:
1. The system creates the recommended index with low visibility (e.g.
`WITH (visibility = 0.05)`), so it is available to ~5% of queries.
2. For each statement that could use the index, we measure performance
when the index is visible vs. when it is not (using the existing
statement statistics + plan fingerprints, or a dedicated A/B
collection path).
3. After a trial period (configurable by time and/or sample count),
the system aggregates results per statement fingerprint:
- If the index meaningfully improves performance overall, promote
it to fully visible.
- If it causes regressions, demote it to fully invisible or drop
it entirely.
- Per-statement hints/overrides could be applied where the index
helps some fingerprints but hurts others.
The decision policy should be conservative — prefer leaving things as
they were if the signal is noisy — and should expose its reasoning via
`crdb_internal` so operators can inspect and override.
**Describe alternatives you've considered**
An alternative experimentation primitive: create the index as fully
*invisible*, then for a sampled fraction of statements, run the query
plan twice — once with only visible indexes, and once allowing the
invisible index — and compare. This avoids any production traffic
actually using the new index until it's promoted, at the cost of extra
work per sampled query and added complexity in the execution path.
The partial-visibility approach is simpler and reuses existing
machinery, but exposes a small fraction of real traffic to the unproven
index. Both approaches are worth prototyping.
Other alternatives:
- Manual operator-driven validation (status quo).
- Offline replay of captured workloads against a shadow cluster — more
accurate but much more expensive and operationally heavy.
**Additional context**
This builds on existing functionality:
- Index recommendations produced by the optimizer.
- Partially visible indexes (`visibility` storage param).
- Statement statistics and plan fingerprints for measuring performance.
Open design questions:
- How long should the trial period be, and how is "enough signal" defined?
- How do we attribute regressions to the new index vs. unrelated workload
changes?
- Should this be opt-in per recommendation, per table, or cluster-wide?
- Interaction with schema changes, backups, and rolling upgrades.
Epic CRDB-64888
Jira issue: CRDB-64889
Contributor guide
Research direction
No implementation files or tests are named. Start by tracing the optimizer's index recommendations, partially visible indexes and their visibility storage parameter, then review statement statistics, plan fingerprints, and the crdb_internal exposure. Done means a conservative, configurable trial can measure outcomes and keep, hide, or drop an index with inspectable reasoning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100