hedgehogqa / hedgehogqa/fsharp-hedgehog

A way to stop shrinking from within property evaluation ?

Open
#316 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
F#
Stars
284
Forks
31
PR merge metrics
No merged PRs in 30d

Description

When writing my first property tests with Hedghog, I bumped into SUT entering an endless loop. I solved this with running the offending function in a new Task and cancelling it on timeout via Thread.Abort and/or CancellationToken. In FsCheck this works fine for reasonable timeout durations. Hedgehog, however, treats timeout as a counterexample and starts trying to shrink it. Often this leads to another timeout. For 5 seconds of timeout and shrink limit of 20 it takes more that a minute. Depending on the property and the system, it might take a lot longer.

A developer debugging a property can handle this by adjusting the timeout value and the shrink limit. A CI server, on the other hand, will waste potentially a lot of time in attempts to shrink a hanging property, if the hanging happened in place where no one expected it. Disabling shrinking or setting the limit low enough seems to be a bad idea, as automatic shrinking is the reason I'm trying Hedgehog in the first place.

Right now, I solved this by detecting if the test is running on a CI server and enabling shrinking based on that. But it would be better if there was a way of telling Hedgehog that in some cases, detected at run time, further shrinking is to be skipped. One way to do this is by using a custom StopShrinkingException, but there may be other ways.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.