hedgehogqa / hedgehogqa/fsharp-hedgehog

Mutable structures not always recreated during shrink

Open
#157 3 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
F#
Stars
284
Forks
31
PR merge metrics
No merged PRs in 30d

Description

If (say) an array is declared before the generator binds in a property computation expression, it doesn't get cleared when shrinking after a test failure. If the declaration is moved after the `let!` then it's ok.

This can produce misleading failure messages where the reported state is a combination of what happened during the shrinkage, not what caused the original failure.

As a (contrived) repro case, this test takes 10 integers and copies the distinct values into an array of int options. It then tests that the `Some` values are the *sorted* distinct values (so it fails).

```fsharp
let copyIntoArray (target : array<_>) source =
source |> Array.iteri (fun i x -> target.[i] <- Some x)

[]
let ``strange`` () =
property {
let buffer = Array.zeroCreate 10

let! input =
Gen.int (Range.constant 1 100)
|> Gen.array (Range.constant 10 10)

// Failure message is sensible if buffer is declared here instead
// let buffer = Array.zeroCreate 10

let actual =
input |> Array.distinct |> copyIntoArray buffer
buffer |> Array.choose id

let expected =
input |> Array.distinct |> Array.sort

printfn "testing %A vs %A" actual expected

test <@ actual = expected @>
} |> Property.check
```
During shrinkage, the `input` array is made shorter but the `buffer` array has items left-over. The output reads:
```
testing [|38; 21; 80; 43; 23; 59; 13; 56; 44; 99|] vs [|13; 21; 23; 38; 43; 44; 56; 59; 80; 99|]
testing [|1; 21; 80; 43; 23; 59; 13; 56; 44; 99|] vs [|1; 13; 21; 23; 43; 44; 56; 59; 80; 99|]
testing [|1; 80; 43; 23; 59; 13; 56; 44; 99; 99|] vs [|1; 13; 23; 43; 44; 56; 59; 80; 99|]
testing [|1; 43; 23; 59; 13; 56; 44; 99; 99; 99|] vs [|1; 13; 23; 43; 44; 56; 59; 99|]
testing [|1; 23; 59; 13; 56; 44; 99; 99; 99; 99|] vs [|1; 13; 23; 44; 56; 59; 99|]
testing [|1; 59; 13; 56; 44; 99; 99; 99; 99; 99|] vs [|1; 13; 44; 56; 59; 99|]
testing [|1; 13; 56; 44; 99; 99; 99; 99; 99; 99|] vs [|1; 13; 44; 56; 99|]
testing [|1; 56; 44; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 44; 56; 99|]
testing [|1; 44; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 44; 99|]
testing [|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 99|]
testing [|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1|]
Hedgehog.FailedException : *** Failed! Falsifiable (after 1 test and 10 shrinks):
[|1; 1; 1; 1; 1; 1; 1; 1; 1; 1|]
NUnit.Framework.AssertionException:

[|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] = [|1|]
false
```
But if the buffer declaration is moved, the shrinkage works properly and the message reads:
```
Hedgehog.FailedException : *** Failed! Falsifiable (after 1 test and 24 shrinks):
[|1; 1; 1; 1; 1; 1; 3; 2; 1; 1|]
NUnit.Framework.AssertionException:

[|1; 3; 2|] = [|1; 2; 3|]
false
```

Even if this is by-design behaviour, it might be worth noting it in the docs

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.