nick8325 / nick8325/quickcheck

Use results from previous tests to inform shrinks

Open
#401 12 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Haskell
Stars
790
Forks
130
Avg merge
16h 41m
Merged PRs (30d)
2

Description

This is a somewhat esoteric feature: it might just be us that want it, but thought I'd ask because it seems like it might be simple to add.

We're using quickcheck to compare hardware implementations (https://github.com/CTSRD-CHERI/QuickCheckVEngine/), and would like to use the outputs from runs to decide how to shrink.

To be more specific: we generate a test, which is a sequence of instructions, we call out to the hardware implementations that annotate the sequences of instructions with the results from running the instructions, then we check that the annotations match.

It would be cool if we could input the annotated traces into the shrink function, rather than the unannotated ones. I believe this could be achieved by providing a slightly different API that splits up the annotation of a test from evaluating whether it was a pass or fail, i.e. a version of forAllShrink with the type changed from:

forAllShrink :: (Show a, Testable prop) => Gen a -> (a -> [a]) -> (a -> prop) -> Property

to

forAllShrink :: (Show a, Testable prop) => Gen a -> (a -> b) -> (b -> [a]) -> (b -> prop) -> Property

I'd be interested to hear if you have any thoughts on how tricky this would be (possibly complicated by b being an IO) and whether we've missed a way to do it using the existing API.

Thanks for maintaining a great library!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing forAllShrink API and its implementation, then trace how generated values move through shrinking and property evaluation. Assess how annotated traces, including an effectful annotation step, could fit the requested API. Done means reaching an agreed design and validating it with coverage for shrinking based on prior results.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.