inconsistent test count reported in Jenkins if tests are flaky
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 55
- Forks
- 39
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 7
Description
The nightly jobs reruns failed test with ctest --rerun-failed. But since this run overwrites the generated Test.xml Jenkins will report a different number of test because tests which passed in the initial run are not included in the new result file anymore. I don't think there is much we can do (except parsing the result file before and after and implementing our own merge strategy). I have reported the problem upstream, maybe it gets addressed in CTest directly: https://gitlab.kitware.com/cmake/cmake/issues/16314
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the nightly job configuration that runs ctest --rerun-failed and inspect how Test.xml is passed to Jenkins. Compare the initial and rerun result behavior, then check upstream CMake issue 16314 before deciding whether this repository has an actionable workaround. Done means Jenkins reports a consistent test count without breaking flaky-test reruns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake
- Domain
- ci-cd, infrastructure, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100