IntersectMBO / IntersectMBO/plutus

Fix and add new tests for controlling distribution of generated values

Open
#6,332 0 comments 0 reactions 0 assignees View on GitHub
Good first issue Low priority status: triaged tech debt Test
Dominant language
Haskell
Stars
1.6k
Forks
508
Avg merge
3d 10h
Merged PRs (30d)
22

Description

We have a number of tests showing the distribution of generated values, for example

```haskell
test_listDistribution :: Int -> ([()] -> Gen [[()]]) -> Int -> TestTree
test_listDistribution numRuns split n =
testProperty ("for a list of length " ++ show n) $
withMaxSuccess numRuns . forAll (split $ replicate n ()) $ \aSplit ->
label (show $ map length aSplit) True

test_multiSplitDistribution :: TestTree
test_multiSplitDistribution =
testGroup "distribution of values generated by"
[ testGroup "multiSplit1"
[ test_listDistribution 10000 (coerce $ multiSplit1 @()) 1
, test_listDistribution 10000 (coerce $ multiSplit1 @()) 2
, test_listDistribution 10000 (coerce $ multiSplit1 @()) 3
, test_listDistribution 10000 (coerce $ multiSplit1 @()) 4
, test_listDistribution 10000 (coerce $ multiSplit1 @()) 5
]
, testGroup "multiSplit0"
[ test_listDistribution 1000 (multiSplit0 0.1) 1
, test_listDistribution 1000 (multiSplit0 0.05) 2
, test_listDistribution 1000 (multiSplit0 0.01) 3
]
]
```

but we don't have any tests to ensure that the distribution is sensible, you have to eyeball it.

And when we do have such tests, they usefulness is quite limited:

```haskell
schnorrSecp256k1Prop :: PropertyT IO ()
schnorrSecp256k1Prop = do
testCase <- forAllWith ppShow genSchnorrCase
cover 5 "malformed verification key" . is (_ShouldError . _BadVerKey) $ testCase
cover 5 "malformed signature" . is (_ShouldError . _BadSignature) $ testCase
cover 5 "mismatch of signing key and verification key" . is (_Shouldn'tError . _WrongVerKey) $ testCase
cover 5 "mismatch of message and signature" . is (_Shouldn'tError . _WrongSignature) $ testCase
cover 5 "happy path" . is (_Shouldn'tError . _AllGood) $ testCase
runTestDataWith def testCase id VerifySchnorrSecp256k1Signature

ed25519Prop :: BuiltinSemanticsVariant DefaultFun -> PropertyT IO ()
ed25519Prop semvar = do
testCase <- forAllWith ppShow genEd25519Case
cover 5 "malformed verification key" . is (_ShouldError . _BadVerKey) $ testCase
cover 5 "malformed signature" . is (_ShouldError . _BadSignature) $ testCase
cover 5 "mismatch of signing key and verification key" . is (_Shouldn'tError . _WrongVerKey) $ testCase
cover 5 "mismatch of message and signature" . is (_Shouldn'tError . _WrongSignature) $ testCase
cover 5 "happy path" . is (_Shouldn'tError . _AllGood) $ testCase
runTestDataWith semvar testCase id VerifyEd25519Signature
```

Those `cover 5` should be `cover 15` or similar.

Contributor guide

Open the contributing guide

Research direction

Find the tests containing test_listDistribution, test_multiSplitDistribution, schnorrSecp256k1Prop, and ed25519Prop, then read how generated-value distributions and cover requirements are currently checked. Add tests that verify the distributions are sensible and adjust the shown cover thresholds where appropriate; done means the checks no longer require eyeballing and the relevant test suite passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.