cockroachdb / cockroachdb/cockroach
roachtest: API for performance-based assertions
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
A _fluent_ API for performance-based assertions is an integral part of the proposal for the performance isolation testing framework [1]. It's also a prerequisite for upgrade testing [2], and failure injection [3]. The basic idea is that any of the aforementioned frameworks must have a reliable way to query application and system metrics. Some examples include asserting on memory utilization before and after a failure injection in order to detect memory leak(s), asserting on background workload throughput and latency _during_ cluster upgrade(s), asserting on a host of performance/health metrics during cluster maintenance, etc.
Currently, there are only a handful of roachtests which assert on performance [4], [5]. Both examples query prometheus to extract (and aggregate) application or system metrics, using some (untested) boilerplate. The goal of the API is to make these queries _fluent_ and reliable; i.e., performance-based assertions should be as simple as something like,
```
metrics.assert_that("http_requests_total")
.for_service("auth-service")
.over_last("5m")
.has_rate_at_least(500)
```
[1] https://docs.google.com/document/d/1dsxJajowrenMn0VcvaE1XHK2-Iy-rT7twzxN_5mDhnI/edit?usp=sharing
[2] https://cockroachlabs.atlassian.net/browse/CRDB-25486
[3] https://docs.google.com/document/d/1iwewT4V_HxXN9Xa-bwngDmwxVqZP3aL1IFvIonjhFWg/edit?usp=sharing
[4] https://github.com/cockroachdb/cockroach/blob/61700b1044e25a8ea71e4c07a1681be521312760/pkg/cmd/roachtest/tests/drt.go#L105
[5] https://github.com/cockroachdb/cockroach/blob/61700b1044e25a8ea71e4c07a1681be521312760/pkg/cmd/roachtest/tests/admission_control_disk_bandwidth_overload.go#L183
Jira issue: CRDB-48814
Contributor guide
Assessment
This issue has not been assessed yet.