google / google/xls

Improve fuzzer automatic triage

Open
#426 0 comments 0 reactions 0 assignees View on GitHub
fuzz
Dominant language
C++
Stars
1.9k
Forks
283
Avg merge
2d 10h
Merged PRs (30d)
135

Description

Currently triaging failures in the fuzzer is a manual process which can be tedious as the fuzzer may find 10's of failures in a run. The fuzzer should be able to categorize the underlying cause in most cases. A mechanism might be to add an enum cause to the SampleError exception class. So a cause can be added at exception raising time. A reasonable place to emit the cause is in the exception message itself.

Possible enum values for causes:

```python
class FailureCause(enum):
MISMATCH_JIT # Only the JIT (unoptimized and/or optimized) produces a different result
MISMATCH_BAD_OPT # Only the optimized IR (JIT *and* interpereter) produces a different result
MISMATCH_SIM # simulation produced a different result
MISMATCH_DSLX_INTREPRETER # DSLX interpreter produced a different result
MISMATCH_OTHER # Other kind of mismatch
CRASH_IR_CONVERSION # IR conversion returned a non-zero status
CRASH_OPT # opt_main returned a non-zero status
CRASH_CODEGEN # codegen_main returns a non-zero status
CRASH_SIM # simulate_module_main returned a non-zero status
UNKNOWN # unkown error occured
```

Another nice thing would be a triage tool which could walk a directory of crashers, rerun each one to see if it still fails, and then move failing ones to a directory named with the cause and move non-failing ones to a does not repro directory.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.