hedgehogqa / hedgehogqa/fsharp-hedgehog
A way to stop shrinking from within property evaluation ?
- Dominant language
- F#
- Stars
- 284
- Forks
- 31
- PR merge metrics
- No merged PRs in 30d
Description
When writing my first property tests with Hedghog, I bumped into SUT entering an endless loop. I solved this with running the offending function in a new Task and cancelling it on timeout via Thread.Abort and/or CancellationToken. In FsCheck this works fine for reasonable timeout durations. Hedgehog, however, treats timeout as a counterexample and starts trying to shrink it. Often this leads to another timeout. For 5 seconds of timeout and shrink limit of 20 it takes more that a minute. Depending on the property and the system, it might take a lot longer.
A developer debugging a property can handle this by adjusting the timeout value and the shrink limit. A CI server, on the other hand, will waste potentially a lot of time in attempts to shrink a hanging property, if the hanging happened in place where no one expected it. Disabling shrinking or setting the limit low enough seems to be a bad idea, as automatic shrinking is the reason I'm trying Hedgehog in the first place.
Right now, I solved this by detecting if the test is running on a CI server and enabling shrinking based on that. But it would be better if there was a way of telling Hedgehog that in some cases, detected at run time, further shrinking is to be skipped. One way to do this is by using a custom StopShrinkingException, but there may be other ways.
Contributor guide
Assessment
This issue has not been assessed yet.