cockroachdb / cockroachdb/cockroach
roachtest: first-class support for metamorphic test parameters
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We should support metamorphic test parameters in roachtests. There are two aspects to this:
1. Support enabling existing metamorphic test parameters in the `cockroach` binary during roachtests.
* Needs to output the seed, along with which parameters were enabled for which nodes.
* Consider whether we should use the same or different parameters for different nodes: unclear if they're always compatible.
2. Support using metamorphic parameters in roachtests themselves.
* Currently uses one-off parameters: [`EncryptionMetamorphic`](https://github.com/cockroachdb/cockroach/blob/aac830251139eca515ae31053414bb7ab624a115/pkg/cmd/roachtest/registry/encryption.go#L43-L47) and [`MetamorphicLeases`](https://github.com/cockroachdb/cockroach/blob/0df3a03e781cf780f291cd4302d74083011b5bc1/pkg/cmd/roachtest/registry/test_spec.go#L232-L234).
* Should possibly use same/similar infrastructure as unit/integration tests instead.
* Should be possible to add metamorphic parameters across all tests, or ad hoc for an individual test.
* Metamorphic parameters should be able to control at least cluster settings and env vars.
* Roachperf benchmarks must be excluded from metamorphism.
* Individual tests must be able to opt-out of metamorphism, or override it with specific parameters.
Jira issue: CRDB-29210
Contributor guide
Assessment
This issue has not been assessed yet.