Minimizing timeout test cases
- Dominant language
- Shell
- Stars
- 12.6k
- Forks
- 2.9k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 62
Description
In my experience, test cases for timeouts are not minimized. I'm not sure if this is intentional, or a failure of the system to do the minimization. When triggering minimization manually, I get "Minimize task errored out: LibFuzzer minimization failed.". Given how painful it is to minimize these locally, it would be very useful to support this, including setting a reasonable timeout value.
In many cases, timeouts relate directly to the size of the input. In igraph, we encountered cases where it's the specific structure of the input was to blame. Obtaining a small test case is usually necessary to debug these.
Contributor guide
Research direction
Start by tracing the manual minimization path described in the issue and the LibFuzzer failure message. Determine how timeout cases are currently handled, where a minimization timeout would be configured, and what successful minimization of a timeout should produce; the issue names no files or tests.
Written by the indexing model from the issue text.
Assessment
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100