checkstyle / checkstyle/contribution
Regression report displays exceptions inconsistently
- Dominant language
- Java
- Stars
- 57
- Forks
- 168
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 15
Description
Discovered at https://github.com/checkstyle/checkstyle/pull/14791#issuecomment-2053817456
Regression report displays exceptions inconsistently.
Sometimes there is a full stacktrace for an exception:
https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8039a13_2024203306/reports/diff/elasticsearch/index.html#A139
Sometimes there is only a line of the exception name:
https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8039a13_2024203306/reports/diff/elasticsearch/index.html#A1
Both these two cases came from the same report generation:
https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8039a13_2024203306/reports/diff/index.html
---
## To reproduce
1. Create a new branch contains the fix for https://github.com/checkstyle/checkstyle/pull/14791.
2. Create a new branch from the commit before https://github.com/checkstyle/checkstyle/pull/14791 is merged.
3. Prepare the following configuration:
Diff Regression config: https://gist.githubusercontent.com/Lmh-java/95ba8a3850cf6d670b2d65262531fb35/raw/7f17357bf8ec422286181c2d470ed578f8da485d/magic-number-config.xml
project lists: use the default (or only the following line for bare minimum)
```
elasticsearch|git|https://github.com/elastic/elasticsearch|v1.5.2||
```
4. run
```console
groovy diff.groovy --localGitRepo [location of checkstyle project] --baseBranch [branch name without fix] --patchBranch [branch name with fix] --config magic-number-config.xml --listOfProjects test.properties -xm -Dcheckstyle.failsOnError=false --allowExcludes
```
## Some analysis
After investigation, this difference presents before the results are sent to `patch-diff-report-tool` to compare. This difference is presented in `checkstyle-result.xml`. For example:
```xml
```
```xml
```
This means, when the checkstyle report is generated, this difference is already there. I also noticed that the single line stacktrace has `
`(\n) at the very end. This might be the cause of the incorrect termination of the line.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with diff.groovy and the report-generation path before results are sent to patch-diff-report-tool. Compare the two checkstyle-result.xml error entries, especially the trailing \n in the single-line exception, and reproduce the Elasticsearch case using the supplied configuration and command. Done means equivalent exceptions render consistently in the regression report.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy, java
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100