MozillaSecurity / MozillaSecurity/bugmon

Add more meaningful messages when testcase is intermittent

Open
#35 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
11
Forks
2
Avg merge
2h 8m
Merged PRs (30d)
1

Description

We should provide more meaningful error messages when bugs appear to be fixed but are then reproduced during fix bisection (on tip). We could correlate the mismatch between our initial tip attempt and the bisection fix tip attempt. If a mismatch is detected, we should override the bug comment to indicate that the testcase appears to be unreliable.

Testcase crashes using the initial build (mozilla-central 20220129091708-7f00dabac085) but not with tip (mozilla-central 20230127094652-f75c73066b88.)

Unable to bisect testcase (End build crashes!):

    Start: 7f00dabac085e45723ccdea1c7e7e2f61e273b30 (20220129091708)
    End: f75c73066b887c2379158c73c994b5ef95460238 (20230127094652)
    BuildFlags: BuildFlags(asan=False, tsan=False, debug=True, fuzzing=True, coverage=False, valgrind=False, no_opt=False, fuzzilli=False, nyx=False)

Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the testcase fix-bisection flow and the comparison between the initial tip attempt and the bisection fix tip attempt. Verify how mismatches are detected and how the bug comment is generated or overridden; done means intermittent testcases receive a meaningful unreliable-testcase message instead of the generic unable-to-bisect output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.