hedgehogqa / hedgehogqa/fsharp-hedgehog

TimeBox a Property test

Open
#341 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
F#
Stars
284
Forks
31
PR merge metrics
No merged PRs in 30d

Description

I'm building a `TimeboxedPropertyAttribute` for Hedgehog.Xunit. Its documentation:

> Runs tests until the time box is reached or the `tests` parameter is hit (100 by default). If no `size` is provided, a random `size` will be used for each test run.

Raison d'être: I have some tests that take a while to run. During development, I don't care _that_ much about correctness and will therefore timebox said tests. I'll flip them back to normal `property` tests before committing/merging. Since a test might only run once if the timebox is extremely short, I want a random size. (Always testing with `Size=0` isn't great.) This is pretty niche to my workflow, but LMK if any of this sounds interesting and anyone wants it merged upstream.

---

I'm currently implementing it via a hack (but without reflection):

```f#
...
let reportWith config =
Property.forAll invoke
>> match recheck with
| Some (size, seed) -> Property.reportRecheckWith size seed config
| None -> Property.reportWith config
match timeBox with
| Some timeBox ->
let config =
PropertyConfig.defaultConfig
|> PropertyConfig.withTests 1
|> withShrinks shrinks
let tests = tests |> Option.defaultValue 100
let rng = Random()
let report () =
101
|> rng.Next
|> Gen.resize
|> gens
|> reportWith config
let stop = DateTime.UtcNow + timeBox
let rec loop i r =
if DateTime.UtcNow >= stop || i >= tests || isNotOk r then
{ r with Tests = i }
else
report() |> loop (i + 1)
report() |> loop 1
...
```

Basically, I run the test once, see if the timebox is exceeded, and if not run it again, while manually incrementing the test count. A better solution would be to implement it in Hedgehog proper, but this hacky solution suited my needs. I'm willing to add it to Hedgehog if there's interest.

From [this conversation](https://github.com/hedgehogqa/fsharp-hedgehog/discussions/340#discussioncomment-1297254):

> Are you thinking this could be implemented with another combinator?

Yep!

> If so, would this be a property level thing? Or could this be supported with a property config option?

Right now I'm leaning in the direction of something on the level of property, i.e. a sibling of `checkWith`. Some of the qualities of a "TimeBoxedProperty" are at odds with the semantics and types of a normal `checkWith` test, so I don't think I can use the current `PropertyConfig` - I'll need a `TimeBoxedPropertyConfig`. For example, I can imagine a scenario where someone would want to run a test without specifying the testcount and only specifying the timebox. The result will be a test that runs as many times as it can for, say, 1 second. Currently `testcount` defaults to 100. Therefore it isn't enough to simply add an optional `timebox` field to `PropertyConfig`. `Size` also needs to be optional, and when its in the `None` case we'll generate random sizes.

Please feel free to consider this idea outside the scope of Hedgehog. I'm pretty sure Haskell doesn't do this!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.