cockroachdb / cockroachdb/cockroach

testutils: the multi-tenant test configs are too complex

Open
#106,772 1 comment 0 reactions 0 assignees View on GitHub
A-multitenancy A-testing C-bug db-cy-23 T-multitenant
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

xref #76378

### Problem description

There are too many test configs.

- Whether or not to start a tenant server to run the test's queries:
- random (default - desirable)
- always-off
- always-on

- Which tenant server config to use:
- Serverless-like config, using a networked KV tenant config + rate limiting - `StartTenant()`
- Dedicated-like config - `StartSharedProcessTenant()`

All these choices results in (currently) 6 different possible test configurations.

In addition to that complexity, some tests _ignore the config above_ and configure their test tenants manually themselves.

In those tests, it's possible to see at run-time both the test tenant created by the test framework, and the test tenant created by the test code, side-by-side. This is confusing.

### Expected solution

We want to collapse the selection of whether or not to use a test tenant and which server config to use into just 1 knob with just 2 possible values:

- `auto-test-tenant-probabilistically` (default):
- in CCL builds: 30% chance to have no secondary tenant; 30% to use serverless-like config, 30% to use dedicated-like config.
- in OSS builds: 30% chance to have no secondary tenant; 60% chance to use the dedicated-like config.

- `no-auto-test-tenant`: no secondary tenant created by the test framework.

The latter setting can have informational variants (like those defined in #106768) to distinguish cases where there's a bug (yet to be fixed) and from cases when the test truly wishes no automatic behavior.

Jira issue: CRDB-29709

Epic CRDB-31933

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.