adoptium / adoptium/aqa-tests

Generic logic to get get and store benchmark metric

Open
#6,355 6 comments 0 reactions 1 assignee Claimed by @MattyWeee123 View on GitHub
soc_25
Dominant language
HTML
Stars
155
Forks
346
Avg merge
2d 9h
Merged PRs (30d)
36

Description

Extend based on https://github.com/adoptium/aqa-tests/pull/6339/, make checkDacapoH2Metric() and archiveDacapoH2Metric() generic.

On second thought, it may be better to have a separate script (i.e., Python - `benchmarkMetric.py`) to handle this.

- In `benchmarkMetric.py`, it contains a generic function `checkMetric()`. `checkMetric()` takes job console output and reads `benchmarkMetric.json` and runs each regex, and stores into one JSON output.

`benchmarkMetric.json` example:
```
[
{
"Benchmark": "dacapo-eclipse",
"metrics":
{
"dacapo-eclipse" : [
{"regex": "DaCapo.*eclipse\\sPASSED\\sin\\s(\\d*\\.?\\d*)\\smsec"},
{"higherbetter": true},
{"units": "msec"}
]
}
},
{
"Benchmark": "dacapo-h2",
"metrics":
{
"dacapo-h2" : [
{"regex": "DaCapo.*h2\\sPASSED\\sin\\s(\\d*\\.?\\d*)\\smsec"},
{"higherbetter": true},
{"units": "msec"}
]
}
}
]
```

Note: Backslashes must be double-escaped (`\\`) in JSON strings.

Output: JSON file as specified in https://github.com/adoptium/aqa-tests/issues/6327. For example:

```
[
{
"Benchmark": "dacapo-eclipse",
"metrics":
{
"dacapo-eclipse" : [
{"value": "123456"},
{"higherbetter": true},
{"units": "msec"}
]
}
},
{
"Benchmark": "dacapo-h2",
"metrics":
{
"dacapo-h2" : [
{"value": "4689"},
{"higherbetter": true},
{"units": "msec"}
]
}
}
]
```

related: https://github.com/adoptium/aqa-tests/issues/6327

top level issue: https://github.com/adoptium/aqa-tests/issues/6268

@smlambert @LongyuZhang if I missed anything, please feel free to update.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.