adoptium / adoptium/aqa-test-tools

Add Parsers for Various Benchmark Variants from PerfNext, Perffarm & Adopt

Open
#150 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
33
Forks
97
Avg merge
7h 9m
Merged PRs (30d)
5

Description

### Problem Description

Currently, TRSS supports parsing of both PerfNext and Perffarm jobs. While PerfNext builds that use Jenkins are parsed right away after runs are done, Perffarm builds are parsed when Perf Compare is used.

While TRSS does have some parsers, it doesn't have benchmark parsers for many of the benchmark variants that exist on PerfNext and Perffarm, configurations that are used frequently. As a result, we need to manually parse them for the time being.

Furthermore, even if we have parsers for some configs, we parse only some metrics and haven't added support for others, which might be important to get full performance evaluation of a build.

Some of these configs were newly added to launcher tools in order to support new variants and benchmark versions. Also, parsers could have been missed for some of the less popular benchmark variants, something that we should still add in order to extend coverage.

### Benchmark Parsers to Add

**Perffarm & PerfNext:**
- Different Startup and throughput variants for Liberty
- Various ODM variants
- HiBench
- SPECjbb2015
- Any other missing

**Adopt:**

- BumbleBench
- Dacapo
- Idle Micro
- Liberty
- Renaissance

### Proposed Changes

- Add new Perffarm variants to [BenchmarkVariants](https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/master/test-result-summary-client/src/PerfCompare/lib/benchmarkVariantsInfo.js).
- Add new PerfNext variants to [BenchmarkMetricRouter](https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/master/TestResultSummaryService/parsers/BenchmarkMetricRouter.js
) and [BenchmarkMetric](https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/master/TestResultSummaryService/parsers/BenchmarkMetric.js). PerfNext configs can be found [here](https://github.com/AdoptOpenJDK/openjdk-test-tools/tree/master/PerfNext/config/benchmarks/data_simple).

### Assigned Contributors
I'll be working with Dong (@dhlee49) from my team to work on this design.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the benchmark configurations in PerfNext/config/benchmarks/data_simple with existing entries in test-result-summary-client/src/PerfCompare/lib/benchmarkVariantsInfo.js, TestResultSummaryService/parsers/BenchmarkMetricRouter.js, and BenchmarkMetric.js. Review which PerfNext, Perffarm, and Adopt variants and metrics are already covered. Done means the listed benchmark variants can be parsed and their relevant metrics are represented in the appropriate parser mappings.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.