hedgehogqa / hedgehogqa/fsharp-hedgehog
Mutable structures not always recreated during shrink
- Dominant language
- F#
- Stars
- 284
- Forks
- 31
- PR merge metrics
- No merged PRs in 30d
Description
If (say) an array is declared before the generator binds in a property computation expression, it doesn't get cleared when shrinking after a test failure. If the declaration is moved after the `let!` then it's ok.
This can produce misleading failure messages where the reported state is a combination of what happened during the shrinkage, not what caused the original failure.
As a (contrived) repro case, this test takes 10 integers and copies the distinct values into an array of int options. It then tests that the `Some` values are the *sorted* distinct values (so it fails).
```fsharp
let copyIntoArray (target : array<_>) source =
source |> Array.iteri (fun i x -> target.[i] <- Some x)
[]
let ``strange`` () =
property {
let buffer = Array.zeroCreate 10
let! input =
Gen.int (Range.constant 1 100)
|> Gen.array (Range.constant 10 10)
// Failure message is sensible if buffer is declared here instead
// let buffer = Array.zeroCreate 10
let actual =
input |> Array.distinct |> copyIntoArray buffer
buffer |> Array.choose id
let expected =
input |> Array.distinct |> Array.sort
printfn "testing %A vs %A" actual expected
test <@ actual = expected @>
} |> Property.check
```
During shrinkage, the `input` array is made shorter but the `buffer` array has items left-over. The output reads:
```
testing [|38; 21; 80; 43; 23; 59; 13; 56; 44; 99|] vs [|13; 21; 23; 38; 43; 44; 56; 59; 80; 99|]
testing [|1; 21; 80; 43; 23; 59; 13; 56; 44; 99|] vs [|1; 13; 21; 23; 43; 44; 56; 59; 80; 99|]
testing [|1; 80; 43; 23; 59; 13; 56; 44; 99; 99|] vs [|1; 13; 23; 43; 44; 56; 59; 80; 99|]
testing [|1; 43; 23; 59; 13; 56; 44; 99; 99; 99|] vs [|1; 13; 23; 43; 44; 56; 59; 99|]
testing [|1; 23; 59; 13; 56; 44; 99; 99; 99; 99|] vs [|1; 13; 23; 44; 56; 59; 99|]
testing [|1; 59; 13; 56; 44; 99; 99; 99; 99; 99|] vs [|1; 13; 44; 56; 59; 99|]
testing [|1; 13; 56; 44; 99; 99; 99; 99; 99; 99|] vs [|1; 13; 44; 56; 99|]
testing [|1; 56; 44; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 44; 56; 99|]
testing [|1; 44; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 44; 99|]
testing [|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 99|]
testing [|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1|]
Hedgehog.FailedException : *** Failed! Falsifiable (after 1 test and 10 shrinks):
[|1; 1; 1; 1; 1; 1; 1; 1; 1; 1|]
NUnit.Framework.AssertionException:
[|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] = [|1|]
false
```
But if the buffer declaration is moved, the shrinkage works properly and the message reads:
```
Hedgehog.FailedException : *** Failed! Falsifiable (after 1 test and 24 shrinks):
[|1; 1; 1; 1; 1; 1; 3; 2; 1; 1|]
NUnit.Framework.AssertionException:
[|1; 3; 2|] = [|1; 2; 3|]
false
```
Even if this is by-design behaviour, it might be worth noting it in the docs
Contributor guide
Assessment
This issue has not been assessed yet.