checkstyle / checkstyle/contribution

Regression report displays exceptions inconsistently

Open
#858 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
57
Forks
168
Avg merge
1d 11h
Merged PRs (30d)
15

Description

Discovered at https://github.com/checkstyle/checkstyle/pull/14791#issuecomment-2053817456

Regression report displays exceptions inconsistently.

Sometimes there is a full stacktrace for an exception:
https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8039a13_2024203306/reports/diff/elasticsearch/index.html#A139
image

Sometimes there is only a line of the exception name:
https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8039a13_2024203306/reports/diff/elasticsearch/index.html#A1
image

Both these two cases came from the same report generation:
https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8039a13_2024203306/reports/diff/index.html

---
## To reproduce

1. Create a new branch contains the fix for https://github.com/checkstyle/checkstyle/pull/14791.
2. Create a new branch from the commit before https://github.com/checkstyle/checkstyle/pull/14791 is merged.
3. Prepare the following configuration:
Diff Regression config: https://gist.githubusercontent.com/Lmh-java/95ba8a3850cf6d670b2d65262531fb35/raw/7f17357bf8ec422286181c2d470ed578f8da485d/magic-number-config.xml
project lists: use the default (or only the following line for bare minimum)
```
elasticsearch|git|https://github.com/elastic/elasticsearch|v1.5.2||
```
4. run
```console
groovy diff.groovy --localGitRepo [location of checkstyle project] --baseBranch [branch name without fix] --patchBranch [branch name with fix] --config magic-number-config.xml --listOfProjects test.properties -xm -Dcheckstyle.failsOnError=false --allowExcludes
```

## Some analysis

After investigation, this difference presents before the results are sent to `patch-diff-report-tool` to compare. This difference is presented in `checkstyle-result.xml`. For example:
```xml

```
```xml

```

This means, when the checkstyle report is generated, this difference is already there. I also noticed that the single line stacktrace has `
`(\n) at the very end. This might be the cause of the incorrect termination of the line.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with diff.groovy and the report-generation path before results are sent to patch-diff-report-tool. Compare the two checkstyle-result.xml error entries, especially the trailing \n in the single-line exception, and reproduce the Elasticsearch case using the supplied configuration and command. Done means equivalent exceptions render consistently in the regression report.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy, java
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.