cockroachdb / cockroachdb/cockroach
testutils: the multi-tenant test configs are too complex
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
xref #76378
### Problem description
There are too many test configs.
- Whether or not to start a tenant server to run the test's queries:
- random (default - desirable)
- always-off
- always-on
- Which tenant server config to use:
- Serverless-like config, using a networked KV tenant config + rate limiting - `StartTenant()`
- Dedicated-like config - `StartSharedProcessTenant()`
All these choices results in (currently) 6 different possible test configurations.
In addition to that complexity, some tests _ignore the config above_ and configure their test tenants manually themselves.
In those tests, it's possible to see at run-time both the test tenant created by the test framework, and the test tenant created by the test code, side-by-side. This is confusing.
### Expected solution
We want to collapse the selection of whether or not to use a test tenant and which server config to use into just 1 knob with just 2 possible values:
- `auto-test-tenant-probabilistically` (default):
- in CCL builds: 30% chance to have no secondary tenant; 30% to use serverless-like config, 30% to use dedicated-like config.
- in OSS builds: 30% chance to have no secondary tenant; 60% chance to use the dedicated-like config.
- `no-auto-test-tenant`: no secondary tenant created by the test framework.
The latter setting can have informational variants (like those defined in #106768) to distinguish cases where there's a bug (yet to be fixed) and from cases when the test truly wishes no automatic behavior.
Jira issue: CRDB-29709
Epic CRDB-31933
Contributor guide
Assessment
This issue has not been assessed yet.