[Bug]: flaky() retries increment the snapshot counter, so a mismatching snapshot silently self-heals

What Happened

A toMatchSnapshot() assertion inside a test marked flaky() never reports a mismatch. The retry resolves a different snapshot file than the first attempt, so the baseline it was meant to compare against is quietly left behind.

Locally, with a baseline that does not match the current value:

• First run: attempt 1 compares against the baseline and fails. Attempt 2 looks for ..._2.snap, which does not exist, so it is created from the current value. The test ends as incomplete and the suite is green.
• Every run after that: attempt 1 still fails against the baseline, attempt 2 now matches the ..._2.snap it created earlier, and the test passes. The original baseline is never honored again.

On CI the failure is not silent, but it points at the wrong thing. Running the same test with CI=true:

Snapshot is missing at [tests/.pest/snapshots/Feature/FlakySnapshotTest/it_compares_a_snapshot_inside_a_flaky_test__3.snap].
Run Pest with --update-snapshots to create it.

The file it names is one the suite never created, and the value that actually differs is never mentioned.

The combination is not exotic: screenshots are both the assertion people mark flaky() and the assertion whose regression must not be swallowed. That is how we ran into it, on a visual regression suite where four baselines had been drifting for weeks behind a green run.

How to Reproduce

Fresh app, no application code involved:

laravel new pest-flaky-repro --react --pest --database=sqlite --no-authentication --no-node --no-boost
cd pest-flaky-repro

tests/Feature/FlakySnapshotTest.php:

<?php

declare(strict_types=1);

it('compares a snapshot inside a flaky test', function (): void {
    expect('after')->toMatchSnapshot();
})->flaky(3);

Write a baseline that does not match, so the very first attempt fails:

mkdir -p tests/.pest/snapshots/Feature/FlakySnapshotTest
printf 'before' > tests/.pest/snapshots/Feature/FlakySnapshotTest/it_compares_a_snapshot_inside_a_flaky_test.snap

Then:

vendor/bin/pest tests/Feature/FlakySnapshotTest.php   # 1 incomplete, suite green
ls tests/.pest/snapshots/Feature/FlakySnapshotTest/   # a __2.snap now exists, holding "after"
vendor/bin/pest tests/Feature/FlakySnapshotTest.php   # 1 passed

Expected: the test fails on all three attempts, since after never equals the before baseline.

Control: removing ->flaky(3) from the same test, with the same baseline, fails as it should.

For the CI path, delete the generated __2.snap first, then:

CI=true vendor/bin/pest tests/Feature/FlakySnapshotTest.php

Sample Repository

Reproduced on a clean install as described above, so no repository is needed. Happy to publish one if that helps.

It also reproduces outside Laravel entirely, on a bare composer.json whose only dependency is pestphp/pest, with the test in tests/ and the baseline in tests/.pest/snapshots/FlakySnapshotTest/. Same outcome: 1 incomplete, Snapshot created at [..._2.snap]. So nothing in the Laravel integration is involved.

Pest Version

5.1.1, and 5.0.4 behaves identically, so this is not a 5.1 regression.

PHP Version

8.5.8, and 8.4.23 behaves identically.

Operation System

macOS

Notes

The cause looks like a missing reset. SnapshotRepository::$expectationsCounter is static and keyed by test file plus test description, and it is incremented once per snapshot expectation so that several toMatchSnapshot() calls in one test get distinct files. Testable::__callClosure() replays the closure in the same process on a flaky retry without clearing that key, so the counter carries over and attempt 2 asks for __2.

The replay already restores object properties, mock objects and the output buffer before running again; the counter for the current key looks like it belongs to that same cleanup.

One thing that might look like it covers this case but does not: the guard that rethrows instead of retrying when __snapshotChanges is not empty. It protects the case where the first attempt creates a snapshot. Here the first attempt only compares, so the retry proceeds.

Start by reading SnapshotRepository::$expectationsCounter and Testable::__callClosure(), focusing on how flaky retries restore state before replaying the closure. Reproduce the issue with tests/Feature/FlakySnapshotTest.php and its .snap baseline, then verify that the same baseline is used on every attempt and mismatches remain failures across retries.