cockroachdb / cockroachdb/cockroach

roachtest: expand select-probability functionality

Open
#111,864 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-foundations A-testing C-enhancement E-quick-win T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In #105388, we implemented a roachtest CLI parameter to allow users to specify a sampling rate of roachtests. This has allow new features to be added and smoke tested.

The initial implementation ensures that at least one test per prefix is selected, regardless of how small a probability is set, which guarantees solid coverage across a range of tests.

However, the caveat is that even the smallest probability will still run potentially > 100 tests (1 per prefix), which can take hours to run.

We should expose more parameters that allow:
- alternative grouping by owning team
- using strict sampling without minimum per group requirements

e.g.
- `--select-probability 0.1 would return 10% of all tests (after filtering)
- `--select-probability 0.1 --select-group owner --select-group-min ` would return at least 10% of all tests, and a minimum of per owner. (We could also choose to omit allowing to specify a number, in favour of `--select-group-atleast-one`)

Jira issue: CRDB-32103

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.