nick8325 / nick8325/quickcheck
Use results from previous tests to inform shrinks
Nobody has claimed this yet.
- Dominant language
- Haskell
- Stars
- 790
- Forks
- 130
- Avg merge
- 16h 41m
- Merged PRs (30d)
- 2
Description
This is a somewhat esoteric feature: it might just be us that want it, but thought I'd ask because it seems like it might be simple to add.
We're using quickcheck to compare hardware implementations (https://github.com/CTSRD-CHERI/QuickCheckVEngine/), and would like to use the outputs from runs to decide how to shrink.
To be more specific: we generate a test, which is a sequence of instructions, we call out to the hardware implementations that annotate the sequences of instructions with the results from running the instructions, then we check that the annotations match.
It would be cool if we could input the annotated traces into the shrink function, rather than the unannotated ones. I believe this could be achieved by providing a slightly different API that splits up the annotation of a test from evaluating whether it was a pass or fail, i.e. a version of forAllShrink with the type changed from:
forAllShrink :: (Show a, Testable prop) => Gen a -> (a -> [a]) -> (a -> prop) -> Property
to
forAllShrink :: (Show a, Testable prop) => Gen a -> (a -> b) -> (b -> [a]) -> (b -> prop) -> Property
I'd be interested to hear if you have any thoughts on how tricky this would be (possibly complicated by b being an IO) and whether we've missed a way to do it using the existing API.
Thanks for maintaining a great library!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing forAllShrink API and its implementation, then trace how generated values move through shrinking and property evaluation. Assess how annotated traces, including an effectful annotation step, could fit the requested API. Done means reaching an agreed design and validating it with coverage for shrinking based on prior results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100