adoptium / adoptium/aqa-tests

Investigate the diffCoverage story for JDK

Open
#2,297 4 comments 0 reactions 0 assignees View on GitHub
enhancement gsoc
Dominant language
HTML
Stars
155
Forks
346
Avg merge
2d 9h
Merged PRs (30d)
36

Description

From the JDK developer's view it would be nice to know if the change has been covered by existing tests. If not we will need to create a new test to cover the change.

From https://wiki.openjdk.java.net/display/CodeTools/JCov+FAQ we know:
1. run existing test with JCov to obtain result.xml file with coverage information
2. get a patch file for the change
3. use a JCov DiffCoverage utility to see which lines of changed code were not touched by tests

As running with JCov reporting can be very memory intensive we haven't seen any existing build or github action workflow enabled jcov reporting. We'd like to try to automate the process by several steps.
1. setup [a jenkins job](https://ci.adoptopenjdk.net/view/work-in-progress/job/openjdk8_jcov/) to generate coverage information. The job grab the specified JDK, run the specified test (e.g. jdk_math) and archive the jcov result.
Note:

- Currently the job will hit the memory exception for some target. Just update with `-Xmx4g` to see if it help
- the job currently using jtreg to trigger the jcov dynamically, which generates the jcov report. However it delete the mete data `result.xml`. Need to update to use jcov directory and generate the `result.xml`, which can be consumed by diffCoverage.
- Job grab a specified JDK to avoid the building step. So the job need to be updated archive the openjdk source commit|sha information, which is defined as `SOURCE` in release file. `SOURCE=" .:OpenJDK: d5ef2af266:"`

2. **get a patch file for the change**: compare the PR with the build jdk commit|sha
3. **use a JCov DiffCoverage utility**:

```
java -jar jcov.jar DiffCoverage -h
Try "-help-verbose" for more detailed help
Usage:
> java -jar jcov.jar diffcoverage result.xml diff
options:
Manage replacing
-replaceDiff 'string value'
-replaceClass 'string value'
Manage output
-all
Basic options
-help(h, ?)
-help-verbose(hv)
-print-env(env)
-propfile 'string value'
-plugindir 'string value' (By default is: plugins)
-log.file 'string value'
-log.level(log) 'string value' (By default is: )
Example: java -jar jcov.jar diffcoverage -replace src/classes/: result.xml diff
```

Contributor guide

Open the contributing guide

Research direction

Start with the linked Jenkins job openjdk8_jcov and its jtreg/JCov reporting setup. Check how the job preserves result.xml and the JDK SOURCE commit, then review the DiffCoverage command and its patch input. Done means the workflow can produce matching coverage and source metadata and report changed lines not covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.