cockroachdb / cockroachdb/cockroach

sql/schemachanger: -run-all-cumulative does not extend mixed-version coverage in PauseMixedVersion

Open
#175,462 0 comments 0 reactions 0 assignees View on GitHub
A-schema-changer-impl C-investigation O-agent O-schema-testing T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Summary:**

The `PauseMixedVersion` cumulative test family pauses a schema change at a post-commit stage, upgrades the cluster to the latest version, and resumes. All of a fixture's stage cases share one cluster, and the upgrade is permanent — so only the first case to run is genuinely mixed-version. `-run-all-cumulative`, whose purpose is to defeat stage sampling and exercise every stage, therefore cannot produce mixed-version coverage for more than one stage per fixture. We currently have no way to test a pause-and-upgrade at an arbitrary post-commit stage.

Raised by @shghasemi during review of https://github.com/cockroachlabs/cockroach/pull/4751.

**Findings:**

- `pausePrepare` creates a single cluster per fixture, and every stage case generated from that fixture reuses it.
- Each case does `SET CLUSTER SETTING VERSION = ` while the job is paused. Nothing downgrades the cluster afterwards and the cluster is not recreated between cases.
- Consequently, cases 2..N for a fixture plan and execute wholly at the latest version. The "upgrade" is a no-op for them and the case degenerates into plain `Pause`.
- This has been true since the stage cases began sharing a cluster — effective mixed-version coverage has been one stage per fixture all along, regardless of how many cases were generated.
- `PauseMixedVersion` now samples one stage per fixture explicitly rather than generating cases that silently duplicate `Pause`. `-run-all-cumulative` is still honored in the sense that it returns every case, but the extra cases add no mixed-version coverage — they just cost CI time.
- Genuinely covering every stage in mixed-version would require a fresh cluster per stage, which is precisely the cost the shared-cluster design was introduced to avoid.

**Code References:**

- [`pkg/sql/schemachanger/sctest/cumulative.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/schemachanger/sctest/cumulative.go) — `PauseMixedVersion`, `pausePrepare`, and the `SET CLUSTER SETTING VERSION` in `pauseCase`
- [`pkg/sql/schemachanger/sctest/framework.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/schemachanger/sctest/framework.go) — `runAllCumulative` flag and the stage sampling functions

**Next Steps:**

- [ ] Decide whether per-stage mixed-version coverage is worth its cost, or whether one stage per fixture is adequate given that the interesting transition is the upgrade itself rather than the specific stage it lands on.
- [ ] If broader coverage is wanted, evaluate the options: recreate the cluster per stage (correct but expensive), or rotate the sampled stage deterministically across CI runs so coverage accumulates over time at no extra per-run cost.
- [ ] Consider whether `-run-all-cumulative` should warn or be rejected for `PauseMixedVersion`, since it currently implies a coverage guarantee it cannot deliver.

Jira issue: CRDB-68385

Epic CRDB-60540

Contributor guide

Open the contributing guide

Research direction

Read PauseMixedVersion, pausePrepare, pauseCase, and the stage-sampling logic in pkg/sql/schemachanger/sctest/cumulative.go, then inspect runAllCumulative in framework.go. Decide and document the intended coverage policy, implement the selected behavior or warning, and add coverage showing that -run-all-cumulative no longer implies unsupported mixed-version coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.