elm-explorations / elm-explorations/test

Default runs too low

Open
#190 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Elm
Stars
244
Forks
40
Avg merge
1d 23h
Merged PRs (30d)
2

Description

A gripe I've had for a while is that the default runs of 100 is way too low to get decent coverage for most scenarios, and in my experience is pretty low for most test suites.

I recommend usually about 10,000 runs as a base number, then adjust based on desired run time.

I think from a DX perspective having it specified as the number of runs should really be considered as an abstraction leak. Property tests assert that a condition holds for all inputs meeting some criteria; the implementation detail of verifying that assertion is generating a certain number of samples, but the user doesn't necessarily have a great mental model of how many those samples should be (and indeed understanding this requires some fairly non-trivial statistical understanding, as well as knowledge about implementation details of the fuzzers, etc).

Here are some practical suggestions on how to improve this:

1. Let the user specify (wall-clock) time that they want the test suite to run for. This is nice, since for instance in watch mode we might want to prioritise fast iteration time, in CI we often have other jobs running in parallel so we have a pretty good idea how much "spare" time our tests can take.
2. Specify a minimum coverage as a percentage (this would make more sense with #188), i.e. we want to validate a certain percentage of the available input space. (Ergonomically this might be nicer to specify in some smaller unit, like 1/1,000,000 or some such). This is nice in the sense that it directly specifies our certainty of not having a bug :)
3. Have labelling #94 and run enough tests to achieve enough distributions on each label.

---

A separate issue that could be resolved much more quickly (and is also breaking) is that `Test.fuzzWith` expects an absolute number of runs. I think this is un-ergonomic, since it's a value one needs to keep messing with. A nicer design would be as a multiplier of the globally configured value. This can be used both for "this test is super slow, so let's not waste too much time testing it" to "this test has highly variable behaviour, so let's spend a lot of our time testing the input space", but let's the test runner also influence the total number of tests to run.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.