Improve fuzzer automatic triage
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 283
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 135
Description
Currently triaging failures in the fuzzer is a manual process which can be tedious as the fuzzer may find 10's of failures in a run. The fuzzer should be able to categorize the underlying cause in most cases. A mechanism might be to add an enum cause to the SampleError exception class. So a cause can be added at exception raising time. A reasonable place to emit the cause is in the exception message itself.
Possible enum values for causes:
```python
class FailureCause(enum):
MISMATCH_JIT # Only the JIT (unoptimized and/or optimized) produces a different result
MISMATCH_BAD_OPT # Only the optimized IR (JIT *and* interpereter) produces a different result
MISMATCH_SIM # simulation produced a different result
MISMATCH_DSLX_INTREPRETER # DSLX interpreter produced a different result
MISMATCH_OTHER # Other kind of mismatch
CRASH_IR_CONVERSION # IR conversion returned a non-zero status
CRASH_OPT # opt_main returned a non-zero status
CRASH_CODEGEN # codegen_main returns a non-zero status
CRASH_SIM # simulate_module_main returned a non-zero status
UNKNOWN # unkown error occured
```
Another nice thing would be a triage tool which could walk a directory of crashers, rerun each one to see if it still fails, and then move failing ones to a directory named with the cause and move non-failing ones to a does not repro directory.
Contributor guide
Assessment
This issue has not been assessed yet.