adoptium / adoptium/aqa-test-tools
Update Benchmark Versions & Add Profiling Parameters to PerfNext Launcher
- Dominant language
- Jupyter Notebook
- Stars
- 33
- Forks
- 97
- Avg merge
- 7h 9m
- Merged PRs (30d)
- 5
Description
### Problem Description
Currently, all benchmark configs are using older version of benchmarks. We should update them to these latest benchmarks.
| Current Version | Version to Use
-- | -- | --
Liberty | 19.0.0.2 | 19.0.0.9
ODM | 8.8.1 | 8.10.0
SPECjbb2015 | SPECjbb2015GMR | specjbb2015v101_jaxb24
Also, we recently extended the support for profiling tools to ODM from Liberty. In future, this profiling ability would be extended further to other benchmarks. As part of that work, we also added the capability in our scripts to use the profiling parameters passed by the user or to set them to default if nothing's specified.
These profiling parameters should be added to the relevant benchmark configs so that user can tune these params according to their needs.
We should add all these parameters mentioned below and anything else that I might have missed:
1. `PROFILING_TOOL`
2. `PROFILING_JAVA_OPTIONS`
3. `PROFILING_PROFILE_TIME`
4. `PROFILING_SLEEP_TIME`
5. `PERF_SAMPLING_PERIOD`
6. `PERF_EVENT`
### Proposed Work
- [ ] Update benchmark versions
- [ ] Add any new or missing profiling parameters to PerfNext Launcher
Selected jobs are the ones that we care about:
https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/b3510949c0d6f24650f48d393769ec5d10c35c2d/PerfNext/config/benchmarks/data_simple/Liberty.xml#L3
https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/b3510949c0d6f24650f48d393769ec5d10c35c2d/PerfNext/config/benchmarks/data_simple/Liberty.xml#L239
https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/b3510949c0d6f24650f48d393769ec5d10c35c2d/PerfNext/config/benchmarks/data_simple/ODM.xml#L3
https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/b3510949c0d6f24650f48d393769ec5d10c35c2d/PerfNext/config/benchmarks/data_simple/ODM.xml#L27
- [ ] After updating the versions and adding new profiling options, we should test our changes with both ODM and Liberty for all the supported profiling tools for each benchmark. We should make sure to test one (i.e. DT7, any ODM) of the 2 jobs from each benchmark suite extensively (i.e. Run all the 7 currently supported profiling tools as mentioned below). For the other jobs (i.e. DT3, other ODM job), we could do some sanity testing to make sure things work.
**Default Testing:** Parameters (Avoid setting profiling parameters unless required. You might just need to set `PROFILING_TOOL`. Leave the non-profiling options as default.)
**Non-default Testing:** Explicitly set all profiling parameters to some value similar to their default but not the same as default value. For example, if the default value for `PROFILING_SLEEP_TIME` is 60, test it with 30 or something.
**Currently Supported Profiling Tools:**
1. jprof tprof
2. jprof scs
3. jprof callflow
4. jprof calltree
5. jprof rtarcf
6. perf stat
7. perf record
Contributor guide
Research direction
Start with PerfNext/config/benchmarks/data_simple/Liberty.xml and ODM.xml at the linked version and job entries. Compare the listed benchmark versions and profiling parameters with the existing benchmark configurations, then test ODM and Liberty using the supported profiling tools and default or explicit parameter values. Done means the requested versions and options work across the specified jobs.
Written by the indexing model from the issue text.
Assessment
- Domain
- performance, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100