elm-explorations / elm-explorations/test

Optionally report stats about fuzz tests (generation, shrinking, rejections)

Open
#184 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Elm
Stars
244
Forks
40
Avg merge
1d 23h
Merged PRs (30d)
2

Description

Related to #94, which is about classifying generated values and showing stats about those classes.

It might be nice to show stats unrelated to classifications too: for example, here's a doc on [how Hypothesis does things](https://hypothesis.readthedocs.io/en/latest/details.html#test-statistics): there, if you run with cmdline argument `--hypothesis-show-statistics`, you'll see:
```
- during generate phase (0.06 seconds):
- Typical runtimes: < 1ms, ~ 47% in data generation
- 100 passing examples, 0 failing examples, 0 invalid examples
- Stopped because settings.max_examples=100
```
or, if you use something that throws out a value with `invalid` (eg. `filter`, `andThen`, ...),
```
test_even_integers:

- during generate phase (0.08 seconds):
- Typical runtimes: < 1ms, ~ 57% in data generation
- 100 passing examples, 0 failing examples, 12 invalid examples
- Events:
* 51.79%, Retried draw from integers().filter(lambda x: x % 2 == 0) to satisfy filter
* 10.71%, Aborted test because unable to satisfy integers().filter(lambda x: x % 2 == 0)
- Stopped because settings.max_examples=100
```
or, if you use an `event` function to classify a generated value,
```
test_even_integers:

- during generate phase (0.09 seconds):
- Typical runtimes: < 1ms, ~ 59% in data generation
- 100 passing examples, 0 failing examples, 32 invalid examples
- Events:
* 54.55%, Retried draw from integers().filter(lambda x: x % 2 == 0) to satisfy filter
* 31.06%, i mod 3 = 2
* 28.79%, i mod 3 = 0
* 24.24%, Aborted test because unable to satisfy integers().filter(lambda x: x % 2 == 0)
* 15.91%, i mod 3 = 1
- Stopped because settings.max_examples=100
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.