nick8325 / nick8325/quickcheck
hspec's `around` doesn't wrap shrunk quickcheck tests
Nobody has claimed this yet.
- Dominant language
- Haskell
- Stars
- 790
- Forks
- 130
- Avg merge
- 16h 41m
- Merged PRs (30d)
- 2
Description
At first I thought this was a bug in hspec, I opened an issue at https://github.com/hspec/hspec/issues/899. But as things currently stand I'm not sure how hspec could fix it without changes to quickcheck. So I'm posting here too to try to get more eyes.
Short version is that hspec has an around function that lets you wrap every run of a quickcheck test. But if the test fails and shrinks, it doesn't wrap the shrunk attempts. For example:
let hook act = do
putStrLn "before"
act
putStrLn "after"
around_ hook $ it "xxx" $ do
let shr = \case
2 -> [1, 0]
_ -> []
forAllShrink getSize shr $ \someInt -> ioProperty $ do
putStrLn $ "Testing " <> show someInt
someInt `shouldNotBe` 2
before
Testing 0
after
before
Testing 1
after
before
Testing 2
after
Testing 1
Testing 0
We have two successful tests that get wrapped, a failing test that gets wrapped, then two shrinks of the failing test that don't get wrapped.
The way this works is that hspec examples can be of type a -> Property, and hooks can provide the a. a -> Property is essentially a -> QCGen -> Int -> Rose Result, where the root Result is the result of the first test and children are the results of any shrinks. (Even if shrinking is unnecessary, every shrink is in the tree, though not actually evaluated.) But that type can't do what we want, since we have a single a to construct the whole tree. Currently it essentially calls hook $ \a -> reduceRose (r a), which means the function passed to the hook
- Does the IO necessary to get the root
Resultof the tree - Does the IO necessary to get the IO necessary to get any immediate child
Results of the tree - Does not get any child
Results of the tree.
Which gives us the behavior observed, that the hook wraps the initial test but not any shrinks.
I don't see any way to get around this with the types as they are. Switching to QCGen -> Int -> Rose (a -> IO Result) might work, but it feels like that would need large changes to both quickcheck and hspec, and I can't rule out that it would fail for some other reason.
Any suggestions?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the example with hspec's around_ and inspect QuickCheck's Property, QCGen, Int, and Rose Result flow. The issue does not identify a file, test, or agreed implementation; completion would require a decided design that wraps shrink attempts and corresponding regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100