cockroachdb / cockroachdb/cockroach

mixedversion: support pausing/resuming workloads for specific versions

Open
#168,535 2 comments 0 reactions 0 assignees View on GitHub
A-testeng-infra C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Motivation
There may be a situation in which we want to intentially stop a workload from running in a mixed-version test. An example case is discussed in this thread https://cockroachlabs.slack.com/archives/C023S0V4YEB/p1776196142598319. This use case seems narrow, thus this is a low priority enhancement, but is a current gap in the framework, so documenting it here. General recommendation for precise workload control is for the test author to write custom hooks to invoke / start / stop the workload binary, which a few tests currently do today. Some examples (as of 25.3 / 25.4):
```
multi-region/mixed-version [test-eng] randomized,timeout: 36h0m0s
multitenant-upgrade [server] randomized,timeout: 5h0m0s
rebalance/by-load/leases/mixed-version [kv] randomized
rebalance/by-load/replicas/mixed-version [kv] randomized
schemachange/mixed-versions [sql-foundations] randomized
schemachange/mixed-versions-compat [sql-foundations]
schemachange/secondary-index-multi-version [sql-foundations] randomized
```

## Summary

The `mixedversion.Test.Workload` helper currently starts the workload via `BackgroundCommand` during the test setup stage and runs it continuously for the entire test duration. There is no framework mechanism to pause or stop the workload after this point.

In theory, under some conditions, the framework could have the workload stop. For the issue linked above in the thread, this case is when the cluster is on a specific version. The reason being that a cluster setting added here https://github.com/cockroachdb/cockroach/pull/155656 (and https://github.com/cockroachdb/cockroach/pull/154916) was not backported to 25.1, and as of writing this issue 25.1 is EoS so that cluster setting cannot be backported. This is a problem in the above thread because without this cluster setting, when the cluster is upgraded to `25.1`, the cluster may experience high cpu usage eventually causing the test to fail.

For this particular case, workarounds are
* setting `MinimumSupportedVersion` to `25.2`, and using custom workload hooks so the workload doesn't start until the cluster is at at >= 25.2. (some combination of `OnStartup`, `InMixedVersion`, and `AfterUpgradeFinalized` hooks)
* OR setting `MinimumBootstrapVersion` to 25.2 to avoid 25.1, but this also avoids 24.3 which as of writing this issue is still supported
* OR setting `WithSkipVersionProbability` to `1`, but this would limit upgrade paths to only `24.3 --> 25.2` so some valid coverage would be lost

While the first workaround works, perhaps the framework could introduce additional helper hooks to control the workload binary.

## Current behavior

`Workload` schedules the workload as a background command that runs for the full test:

```go
func (t *Test) Workload(
name string, node option.NodeListOption, initCmd, runCmd *roachtestutil.Command,
) StopFunc {
...
t.BackgroundCommand(fmt.Sprintf("%s workload", name), node, runCmd)
```

## Potential Enhancement Behavior

The `Workload` helper (or a new option/variant) should support workload lifecycle management for managing the workload when the cluster is on a specific version, or a generalized condition the test author could pass. Behaviors could include:

- A `MinimumWorkloadVersion` option that automatically pauses the workload when the cluster is running a version below the specified threshold
- A callback-based approach where the test can decide whether to run the workload based on the current cluster state
- Start/stop signals tied to upgrade stages

Jira issue: CRDB-62989

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.