MozillaSecurity / MozillaSecurity/bugmon
Add more meaningful messages when testcase is intermittent
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11
- Forks
- 2
- Avg merge
- 2h 8m
- Merged PRs (30d)
- 1
Description
We should provide more meaningful error messages when bugs appear to be fixed but are then reproduced during fix bisection (on tip). We could correlate the mismatch between our initial tip attempt and the bisection fix tip attempt. If a mismatch is detected, we should override the bug comment to indicate that the testcase appears to be unreliable.
Testcase crashes using the initial build (mozilla-central 20220129091708-7f00dabac085) but not with tip (mozilla-central 20230127094652-f75c73066b88.)
Unable to bisect testcase (End build crashes!):
Start: 7f00dabac085e45723ccdea1c7e7e2f61e273b30 (20220129091708)
End: f75c73066b887c2379158c73c994b5ef95460238 (20230127094652)
BuildFlags: BuildFlags(asan=False, tsan=False, debug=True, fuzzing=True, coverage=False, valgrind=False, no_opt=False, fuzzilli=False, nyx=False)
Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the testcase fix-bisection flow and the comparison between the initial tip attempt and the bisection fix tip attempt. Verify how mismatches are detected and how the bug comment is generated or overridden; done means intermittent testcases receive a meaningful unreliable-testcase message instead of the generic unable-to-bisect output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100