pestphp / pestphp/pest

Tia: coverage cache downloaded from another machine empties the coverage report — no per-file rows, bogus Total: 100.0%, --min passes

Open
#1,796 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
11.7k
Forks
538
Avg merge
4d 11h
Merged PRs (30d)
8

Description

Summary

When a TIA baseline recorded on one machine is downloaded and replayed on another (pest()->tia()->baselined()), the coverage cache (coverage.bin.gz) shipped inside the baseline artifact carries the recording machine's absolute paths. After CoverageMerger merges it with the local run's coverage, the final report contains two path roots. The resulting report has zero file nodes, so Pest prints an empty per-file table and Total: 100.0 % — and --coverage --min=X passes regardless of the real coverage. The check fails open, with no warning.

Environment

  • pest v5.0.1, phpunit 13.2.4, phpunit/php-code-coverage 14.2.3
  • Recording machine: GitHub Actions ubuntu-latest (project at /home/runner/work/<repo>/<repo>)
  • Consuming machine: macOS, PHP 8.4.13, xdebug 3.4.7 (XDEBUG_MODE=coverage), project at /Users/<user>/...

Reproduction

  1. On CI, record a baseline with coverage and publish it via the documented flow:
    run: ./vendor/bin/pest --parallel --tia --coverage --fresh
    # upload the directory printed by `pest --baseline` as the pest-tia-baseline artifact
    
  2. On a developer machine (different absolute project path), with pest()->tia()->baselined() in tests/Pest.php:
    XDEBUG_MODE=coverage ./vendor/bin/pest --parallel --tia --coverage --min=68.9
    
    The baseline (including coverage.bin.gz) is downloaded, tests replay, and the output is:
    Tests:    1 skipped, 2619 passed (8738 assertions, 14 uncached, 2606 replayed)
    
    ────────────────────────────────────────────────────────────
                                                  Total: 100.0 %
    
    No per-file rows, bogus 100.0 % total, --min satisfied. Exit code 0.

What we observed inside the merged cache

Unserializing the local coverage.bin.gz after one replay:

  • lineCoverage() contained 2558 files: 1285 under the local root (/Users/...) and 1273 under the recording runner's root (/home/runner/work/...);
  • getReport() on that object produced 0 File nodes (631 Directory nodes), numberOfExecutableLines() === 0, and an empty total percentage;
  • after dropping the foreign-root entries from the data and rebuilding, the report came back to life: 1274 File nodes, 26 061 executable lines, a real total.

The empty report follows from SebastianBergmann\CodeCoverage\Node\Builder::addItems(): it reconstructs each filename from the common path of the whole data set and silently skips any file whose reconstructed path fails is_file(). With two disjoint roots in the data, every file is skipped. Pest's Coverage::report() then iterates zero File nodes (empty table) and percentageOfExecutedLines() over 0/0 renders as 100.0 %, which --min happily accepts.

Root cause

The serialized CodeCoverage cache is machine-specific (absolute paths baked into the data), but:

  • BaselineSync ships it across machines — the downloaded artifact's coverage.bin.gz is written verbatim into local state (BaselineSync::download()state->write(Tia::KEY_COVERAGE_CACHE, $payload['coverage']));
  • CoverageMerger::applyIfMarked() merges it with the local report without rebasing the cached paths to the local project root.

Suggested direction

Any of these would fix the failure mode, in decreasing order of usefulness:

  1. Rebase the cached coverage paths to the consumer's project root before merging (store the recording root alongside the cache, or derive it from the graph);
  2. Don't ship coverage.bin.gz in the baseline artifact / don't import it on fetch — let each machine build its own coverage baseline;
  3. At minimum, detect a foreign root in the cache and discard it with a warning instead of producing an empty report that passes --min at a fake 100.0 %.

Related: #1792 fixes a separate defect in the same merge path (stale cached entries of changed files deflating merged totals). The two compose: cross-machine caches fail open (this issue), same-machine stale caches fail closed (#1792).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with BaselineSync::download() and CoverageMerger::applyIfMarked(), then trace how the downloaded coverage.bin.gz paths reach Coverage::report(). Reproduce the documented TIA baseline flow with different project roots and verify that a foreign-root cache no longer produces an empty report or a false 100.0% result that satisfies --min.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.