cockroachdb / cockroachdb/cockroach

roachtest: API for performance-based assertions

Open
#143,404 1 comment 1 reaction 1 assignee Assigned to @srosenberg View on GitHub
A-testeng-perf C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

A _fluent_ API for performance-based assertions is an integral part of the proposal for the performance isolation testing framework [1]. It's also a prerequisite for upgrade testing [2], and failure injection [3]. The basic idea is that any of the aforementioned frameworks must have a reliable way to query application and system metrics. Some examples include asserting on memory utilization before and after a failure injection in order to detect memory leak(s), asserting on background workload throughput and latency _during_ cluster upgrade(s), asserting on a host of performance/health metrics during cluster maintenance, etc.

Currently, there are only a handful of roachtests which assert on performance [4], [5]. Both examples query prometheus to extract (and aggregate) application or system metrics, using some (untested) boilerplate. The goal of the API is to make these queries _fluent_ and reliable; i.e., performance-based assertions should be as simple as something like,

```
metrics.assert_that("http_requests_total")
.for_service("auth-service")
.over_last("5m")
.has_rate_at_least(500)
```

[1] https://docs.google.com/document/d/1dsxJajowrenMn0VcvaE1XHK2-Iy-rT7twzxN_5mDhnI/edit?usp=sharing
[2] https://cockroachlabs.atlassian.net/browse/CRDB-25486
[3] https://docs.google.com/document/d/1iwewT4V_HxXN9Xa-bwngDmwxVqZP3aL1IFvIonjhFWg/edit?usp=sharing
[4] https://github.com/cockroachdb/cockroach/blob/61700b1044e25a8ea71e4c07a1681be521312760/pkg/cmd/roachtest/tests/drt.go#L105
[5] https://github.com/cockroachdb/cockroach/blob/61700b1044e25a8ea71e4c07a1681be521312760/pkg/cmd/roachtest/tests/admission_control_disk_bandwidth_overload.go#L183

Jira issue: CRDB-48814

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.