cockroachdb / cockroachdb/cockroach

roachtest: allow roachtests to opt out of specific metamorphic constants

Open
#113,164 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-foundations C-enhancement O-roachtest T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

As of #111949, roachtests are now randomly run with runtime assertions and metamorphic constants enabled. Some roachtests make assumptions about the invariance of certain metamorphic constants.

One example is the `asyncpg` test which expects `multiple_active_portals_enabled=false` and expects failures on tests that try and use multiple portals. However, if metamorphic constants are enabled, it is possible for multiple portals to be enabled and those tests will unexpectedly pass.

Currently to fix this, we opt out those roachtest of metamorphic constants and runtime assertions completely by using the standard binary.

https://github.com/cockroachdb/cockroach/blob/1d6e2db725167afd308dc96c2d57e4bd27246064/pkg/cmd/roachtest/tests/split.go#L430

However, this is just a temporary fix as we still want to run these tests with metamorphic constants and runtime assertions. Instead, we should allow tests to specify a metamorphic constants exclusions list. If a constant is specified, that metamorphic constant should be disabled for that test only. This would let us disable metamorphic constants that break tests on a test by test basis.

One particular challenge to this is the fact that metamorphic constants are evaluated on initialization of the cockroach binary. This makes it non trivial to preemptively disable them from the test runner.

Jira issue: CRDB-32784

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.