nick8325 / nick8325/quickcheck

hspec's `around` doesn't wrap shrunk quickcheck tests

Open
#413 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Haskell
Stars
790
Forks
130
Avg merge
16h 41m
Merged PRs (30d)
2

Description

At first I thought this was a bug in hspec, I opened an issue at https://github.com/hspec/hspec/issues/899. But as things currently stand I'm not sure how hspec could fix it without changes to quickcheck. So I'm posting here too to try to get more eyes.

Short version is that hspec has an around function that lets you wrap every run of a quickcheck test. But if the test fails and shrinks, it doesn't wrap the shrunk attempts. For example:

  let hook act = do
        putStrLn "before"
        act
        putStrLn "after"
  around_ hook $ it "xxx" $ do
    let shr = \case
          2 -> [1, 0]
          _ -> []
    forAllShrink getSize shr $ \someInt -> ioProperty $ do
      putStrLn $ "Testing " <> show someInt
      someInt `shouldNotBe` 2
before     
Testing 0
after
before
Testing 1
after
before
Testing 2
after
Testing 1
Testing 0

We have two successful tests that get wrapped, a failing test that gets wrapped, then two shrinks of the failing test that don't get wrapped.

The way this works is that hspec examples can be of type a -> Property, and hooks can provide the a. a -> Property is essentially a -> QCGen -> Int -> Rose Result, where the root Result is the result of the first test and children are the results of any shrinks. (Even if shrinking is unnecessary, every shrink is in the tree, though not actually evaluated.) But that type can't do what we want, since we have a single a to construct the whole tree. Currently it essentially calls hook $ \a -> reduceRose (r a), which means the function passed to the hook

  • Does the IO necessary to get the root Result of the tree
  • Does the IO necessary to get the IO necessary to get any immediate child Results of the tree
  • Does not get any child Results of the tree.

Which gives us the behavior observed, that the hook wraps the initial test but not any shrinks.

I don't see any way to get around this with the types as they are. Switching to QCGen -> Int -> Rose (a -> IO Result) might work, but it feels like that would need large changes to both quickcheck and hspec, and I can't rule out that it would fail for some other reason.

Any suggestions?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the example with hspec's around_ and inspect QuickCheck's Property, QCGen, Int, and Rose Result flow. The issue does not identify a file, test, or agreed implementation; completion would require a decided design that wraps shrink attempts and corresponding regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.