pestphp / pestphp/pest

[Bug]: TIA records a `match` dispatch line as executable-with-zero-hits, so a replayed `--coverage` run is one line short

Open
#1,828 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
11.7k
Forks
538
Avg merge
4d 11h
Merged PRs (30d)
8

Description

What happened

On a replayed TIA run, the coverage report shows one line as uncovered that is genuinely covered, which fails a strict threshold on a suite that is actually at 100%.

$ pest --coverage --exactly=100.0
  Tests:  1093 passed (3062 assertions, 1093 replayed)
  Services/WhatsApp/WhatsAppInboundService ............ 134 / 98.8%
  Total: 99.9 %
  FAIL  Code coverage not exactly 100.0 %, currently 99.9 %.

Line 134 is the dispatch line of a match expression whose every arm is covered:

private function body(WhatsAppMessageType $type, array $message): ?string
{
    return match ($type) {                    // 134 — reported uncovered
        WhatsAppMessageType::Text => WhatsAppValue::nullableString($message, 'text.body'),        // 135
        WhatsAppMessageType::Image => WhatsAppValue::nullableString($message, 'image.caption'),   // 136
        WhatsAppMessageType::Video => WhatsAppValue::nullableString($message, 'video.caption'),   // 137
        WhatsAppMessageType::Document => WhatsAppValue::nullableString($message, 'document.caption'), // 138
        WhatsAppMessageType::Reaction => WhatsAppValue::nullableString($message, 'reaction.emoji'),   // 139
        WhatsAppMessageType::Interactive => WhatsAppValue::nullableString($message, 'interactive.button_reply.title') // 140
            ?? WhatsAppValue::nullableString($message, 'interactive.list_reply.title'),           // 141
        default => null,                      // 142
    };
}
Reproduction

Two commands, same tree, no edits in between:

$ pest --tia --fresh --coverage     # records the graph, runs all 1093 for real
  Total: 100.0 %

$ pest --coverage                   # replays that same freshly-recorded graph
  Total: 99.9 %                     # 1093 replayed

pest --no-tia --coverage also reports 100.0%. So only the replay is short, and it is short against a graph recorded moments earlier — this is not a stale graph.

The recorded snapshot is what is wrong

Unserialising coverage.bin.gz from the TIA storage directory (pest --baseline) shows the snapshot itself holds line 134 as executable with an empty test list, while every arm below it has hits:

126  7 tests                    return true;
132  not executable             private function body(...): ?string
134  EXECUTABLE BUT UNCOVERED   return match ($type) {
135  9 tests                    WhatsAppMessageType::Text => ...
136  1 tests                    WhatsAppMessageType::Image => ...
137  1 tests                    WhatsAppMessageType::Video => ...
138  1 tests                    WhatsAppMessageType::Document => ...
139  1 tests                    WhatsAppMessageType::Reaction => ...
140  1 tests                    WhatsAppMessageType::Interactive => ...
142  9 tests                    default => null,

That state cannot arise from real execution: an arm cannot run without the dispatch line running. So the loss happens while recording the snapshot, and the replay is faithfully reporting bad data.

Probe used:

$dir = /* output of `pest --baseline` */;
$coverage = unserialize(gzdecode(file_get_contents($dir.'/coverage.bin.gz')));
$lines = $coverage->getData()->lineCoverage();
// $lines['<file>'][134] === []  → executable, zero hits
Scope

Exactly one line in a 172-file application is affected. For contrast, in the same app and the same snapshot:

  • the other 10 return match lines all record correctly, with 19–58 tests each
  • one return match (true) records as not executable, and is correctly not counted against the total

So the classification of a match dispatch line is inconsistent between files, and wherever it lands on "executable" without a recorded hit, the total loses that line permanently.

Ruled out
  • Not a stale graph — --fresh immediately before the failing replay.
  • Not a missing test — all arms are covered; no test can add a hit the recorder never persisted.
  • Not the multi-line ?? arm (the one structural difference from the other 10 sites): extracting it into its own method and re-recording produced the identical result on the identical line.
Impact

TIA cannot be combined with a strict coverage threshold: the gate fails on a suite that is genuinely at 100%. The only workaround we found is --no-tia on the coverage command, which removes the speedup exactly where it is largest — here the replayed coverage run takes 4.7s against 116s for the full run.

Environment
  • Pest 5.0.3
  • PHP 8.5.8 (both pcov and xdebug loaded; the run sets XDEBUG_MODE=coverage)
  • Laravel 13
  • config: pest()->tia()->locally(); in tests/Pest.php
  • 1093 tests, 172 files under coverage (<source><include><directory>app</directory>)

Happy to run further probes against this snapshot. The project itself is a client codebase so I cannot share the repo, but I can extract any data you need from the recorded graph or the coverage snapshot.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the two commands, then inspect the TIA snapshot from pest --baseline, especially coverage.bin.gz and getData()->lineCoverage() for the match dispatch line. Trace where TIA records executable coverage and compare that line with its covered arms; done means replayed --coverage --exactly=100.0 reports the same total as the fresh run.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.