elm-explorations / elm-explorations/test

Allow testing `∃x: p(x)`: a test passes at least once out of N tries

Open
#235 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Elm
Stars
244
Forks
40
Avg merge
1d 23h
Merged PRs (30d)
2

Description

Normal fuzz tests check that a fuzzer passes the test _every time_ (`∀x: p(x)`).
Sometimes it's helpful to test whether a fuzzer passes the test _at least once_ (`∃x: p(x)`).

API sketch (with bad naming):

```elm
Test.fuzz : Fuzzer a -> String -> (a -> Expectation) -> Test
Test.fuzzCanPass : Fuzzer a -> String -> (a -> Expectation) -> Test
```

This can be inefficiently done in userspace like this:

```elm
{-| Will run the fuzzer up to 100 times and check if it passes the test at least once.
-}
canPass : Fuzzer a -> (a -> Bool) -> Expectation
canPass fuzzer pred =
let
tries =
100

go n nLeft =
if nLeft <= 0 then
Expect.fail <| "Expected the fuzzer to pass the test at least once out of " ++ String.fromInt tries ++ " tries."

else
let
seenItPass =
Fuzz.examples n fuzzer
|> List.any pred
in
if seenItPass then
Expect.pass

else
go (n + 1) (nLeft - n)
in
go 1 tries
```

since the exposed API lacks the ability to run a _single_ fuzzer with a varying seed. And it's kinda abusing `Fuzz.examples`.

The library would be able to stop fuzzing as soon as it sees a pass; the userspace version generates in chunks and likely throws some of that work away after it finds a passing value.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.