apache / apache/otava

Extend --magnitude to support specific metrics

Open
#2 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
177
Forks
35
Avg merge
3d 16h
Merged PRs (30d)
6

Description

Users report wanting to filter different metrics at different magnitudes (e.g. p99 vs p50). However, the `--magnitude` argument filters across all metrics.

One option could be to allow setting multiple `metric=min_magnitude` pairs with the `--magnitude` flag, or exclusively a single float. This would be pretty verbose... Ex:
```bash
hunter ... --magnitude p50=0.05 --magnitude p99=0.2
```

Another option would be setting the `min_magnitude` in the config for each test's metric, with support for inheritance so each test doesn't need to define it. This option should also support overriding the config.
```yaml
templates:
common_metrics:
metrics:
p50:
scale: 1.0e-6
direction: -1
min_magnitude: 0.05
p99:
scale: 1.0e-6
direction: -1
min_magnitude: 0.2
tests:
local2:
type: csv
file: tests/resources/sample.csv
time_column: time
metrics:
m1:
column: metric1
direction: 1
m2:
column: metric2
direction: -1
min_magnitude: 0.99 # Be able to set min_magnitude for specific tests
attributes: [ commit ]

remote1:
inherit: [common_metrics] # min_magnitude set via inheritance
prefix: "performance_regressions.my_product.%{BRANCH}.test1"

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.