cockroachdb / cockroachdb/cockroach

roachtest: first-class support for metamorphic test parameters

Open
#105,807 5 comments 0 reactions 0 assignees View on GitHub
A-testeng-foundations A-testing C-enhancement quality-friday T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We should support metamorphic test parameters in roachtests. There are two aspects to this:

1. Support enabling existing metamorphic test parameters in the `cockroach` binary during roachtests.
* Needs to output the seed, along with which parameters were enabled for which nodes.
* Consider whether we should use the same or different parameters for different nodes: unclear if they're always compatible.

2. Support using metamorphic parameters in roachtests themselves.
* Currently uses one-off parameters: [`EncryptionMetamorphic`](https://github.com/cockroachdb/cockroach/blob/aac830251139eca515ae31053414bb7ab624a115/pkg/cmd/roachtest/registry/encryption.go#L43-L47) and [`MetamorphicLeases`](https://github.com/cockroachdb/cockroach/blob/0df3a03e781cf780f291cd4302d74083011b5bc1/pkg/cmd/roachtest/registry/test_spec.go#L232-L234).
* Should possibly use same/similar infrastructure as unit/integration tests instead.
* Should be possible to add metamorphic parameters across all tests, or ad hoc for an individual test.
* Metamorphic parameters should be able to control at least cluster settings and env vars.
* Roachperf benchmarks must be excluded from metamorphism.
* Individual tests must be able to opt-out of metamorphism, or override it with specific parameters.

Jira issue: CRDB-29210

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.