elm-explorations / elm-explorations/test
Allow testing `∃x: p(x)`: a test passes at least once out of N tries
- Dominant language
- Elm
- Stars
- 244
- Forks
- 40
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 2
Description
Normal fuzz tests check that a fuzzer passes the test _every time_ (`∀x: p(x)`).
Sometimes it's helpful to test whether a fuzzer passes the test _at least once_ (`∃x: p(x)`).
API sketch (with bad naming):
```elm
Test.fuzz : Fuzzer a -> String -> (a -> Expectation) -> Test
Test.fuzzCanPass : Fuzzer a -> String -> (a -> Expectation) -> Test
```
This can be inefficiently done in userspace like this:
```elm
{-| Will run the fuzzer up to 100 times and check if it passes the test at least once.
-}
canPass : Fuzzer a -> (a -> Bool) -> Expectation
canPass fuzzer pred =
let
tries =
100
go n nLeft =
if nLeft <= 0 then
Expect.fail <| "Expected the fuzzer to pass the test at least once out of " ++ String.fromInt tries ++ " tries."
else
let
seenItPass =
Fuzz.examples n fuzzer
|> List.any pred
in
if seenItPass then
Expect.pass
else
go (n + 1) (nLeft - n)
in
go 1 tries
```
since the exposed API lacks the ability to run a _single_ fuzzer with a varying seed. And it's kinda abusing `Fuzz.examples`.
The library would be able to stop fuzzing as soon as it sees a pass; the userspace version generates in chunks and likely throws some of that work away after it finds a passing value.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.