cockroachdb / cockroachdb/cockroach
sql/schemachanger: -run-all-cumulative does not extend mixed-version coverage in PauseMixedVersion
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Summary:**
The `PauseMixedVersion` cumulative test family pauses a schema change at a post-commit stage, upgrades the cluster to the latest version, and resumes. All of a fixture's stage cases share one cluster, and the upgrade is permanent — so only the first case to run is genuinely mixed-version. `-run-all-cumulative`, whose purpose is to defeat stage sampling and exercise every stage, therefore cannot produce mixed-version coverage for more than one stage per fixture. We currently have no way to test a pause-and-upgrade at an arbitrary post-commit stage.
Raised by @shghasemi during review of https://github.com/cockroachlabs/cockroach/pull/4751.
**Findings:**
- `pausePrepare` creates a single cluster per fixture, and every stage case generated from that fixture reuses it.
- Each case does `SET CLUSTER SETTING VERSION = ` while the job is paused. Nothing downgrades the cluster afterwards and the cluster is not recreated between cases.
- Consequently, cases 2..N for a fixture plan and execute wholly at the latest version. The "upgrade" is a no-op for them and the case degenerates into plain `Pause`.
- This has been true since the stage cases began sharing a cluster — effective mixed-version coverage has been one stage per fixture all along, regardless of how many cases were generated.
- `PauseMixedVersion` now samples one stage per fixture explicitly rather than generating cases that silently duplicate `Pause`. `-run-all-cumulative` is still honored in the sense that it returns every case, but the extra cases add no mixed-version coverage — they just cost CI time.
- Genuinely covering every stage in mixed-version would require a fresh cluster per stage, which is precisely the cost the shared-cluster design was introduced to avoid.
**Code References:**
- [`pkg/sql/schemachanger/sctest/cumulative.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/schemachanger/sctest/cumulative.go) — `PauseMixedVersion`, `pausePrepare`, and the `SET CLUSTER SETTING VERSION` in `pauseCase`
- [`pkg/sql/schemachanger/sctest/framework.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/schemachanger/sctest/framework.go) — `runAllCumulative` flag and the stage sampling functions
**Next Steps:**
- [ ] Decide whether per-stage mixed-version coverage is worth its cost, or whether one stage per fixture is adequate given that the interesting transition is the upgrade itself rather than the specific stage it lands on.
- [ ] If broader coverage is wanted, evaluate the options: recreate the cluster per stage (correct but expensive), or rotate the sampled stage deterministically across CI runs so coverage accumulates over time at no extra per-run cost.
- [ ] Consider whether `-run-all-cumulative` should warn or be rejected for `PauseMixedVersion`, since it currently implies a coverage guarantee it cannot deliver.
Jira issue: CRDB-68385
Epic CRDB-60540
Contributor guide
Research direction
Read PauseMixedVersion, pausePrepare, pauseCase, and the stage-sampling logic in pkg/sql/schemachanger/sctest/cumulative.go, then inspect runAllCumulative in framework.go. Decide and document the intended coverage policy, implement the selected behavior or warning, and add coverage showing that -run-all-cumulative no longer implies unsupported mixed-version coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100