cockroachdb / cockroachdb/cockroach
asim: improve assertion syntax
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Data-driven tests under `pkg/kv/kvserver/asim/tests/testdata/non_rand` use commands such as
```text
assertion stat=qps type=balance ticks=6 upper_bound=1.15
assertion stat=qps type=steady ticks=6 upper_bound=0.05
```
Internally these map to:
- **Balance** → for each tick, `max(store metric) / mean(store metric)` ≤ threshold.
- **Steady** → for each store, over `ticks` history, `|metric – mean|/mean` ≤ threshold.
Shortcomings:
1. Newcomers (author included) struggle with intuitively grasping what these assert statements do and need to go read the code.
2. Authors must remember the cluster tick interval (500 ms) to convert real-time durations → ticks. Various tests never got updated after updating the tick interval, significantly changing the originally envisioned duration.
3. Boiler-plate arg list clutters otherwise readable specs.
## Proposal – two verbs, one pattern
Introduce concise verbs that read like natural language and hide the equation details.
### 1. `spread(metric)` (cross-store balance)
```text
# old: type=balance upper_bound=1.15 ticks=120
assert
spread(qps) <= 15% over 1m
```
*Interpretation*: during the last 1 minute the maximum store QPS is at most 15 % above the mean store QPS at each instant.
### 2. `drift(metric)` (per-store stability)
```text
# old: type=steady upper_bound=0.05 ticks=120
assert
drift(qps) <= 5% over 1m
```
*Interpretation*: for every store, within the last 1 minute its QPS stayed within ±5 % of its own 1-minute average.
Both follow the same template:
```
assert
() <= % over
[can have more assertions here]
```
`` uses Go-style times (e.g. `30s`, `2m`); the harness converts to ticks internally.
## Benefits
* Tests read like the intent they encode — less mental decoding of mean-multiples.
* A single pattern scales; future verbs or units (for example absolute thresholds) can slot into the same skeleton.
## Migration plan
1. Support new syntax in the parser while keeping the old keywords for back-compat.
2. Incrementally convert existing testdata files; trivial mechanical change.
Epic CRDB-56265
Contributor guide
Research direction
Start by locating the parser for assertion commands used by pkg/kv/kvserver/asim/tests/testdata/non_rand and read how the existing balance and steady keywords are handled. Add support for the proposed spread and drift syntax with Go-style durations, preserve the old keywords, and verify that the existing testdata can be migrated without changing assertion behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100