Extend --magnitude to support specific metrics
- Dominant language
- Python
- Stars
- 177
- Forks
- 35
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 6
Description
Users report wanting to filter different metrics at different magnitudes (e.g. p99 vs p50). However, the `--magnitude` argument filters across all metrics.
One option could be to allow setting multiple `metric=min_magnitude` pairs with the `--magnitude` flag, or exclusively a single float. This would be pretty verbose... Ex:
```bash
hunter ... --magnitude p50=0.05 --magnitude p99=0.2
```
Another option would be setting the `min_magnitude` in the config for each test's metric, with support for inheritance so each test doesn't need to define it. This option should also support overriding the config.
```yaml
templates:
common_metrics:
metrics:
p50:
scale: 1.0e-6
direction: -1
min_magnitude: 0.05
p99:
scale: 1.0e-6
direction: -1
min_magnitude: 0.2
tests:
local2:
type: csv
file: tests/resources/sample.csv
time_column: time
metrics:
m1:
column: metric1
direction: 1
m2:
column: metric2
direction: -1
min_magnitude: 0.99 # Be able to set min_magnitude for specific tests
attributes: [ commit ]
remote1:
inherit: [common_metrics] # min_magnitude set via inheritance
prefix: "performance_regressions.my_product.%{BRANCH}.test1"
```
Contributor guide
Assessment
This issue has not been assessed yet.