Automattic / Automattic/wp-codebox

PHPUnit runner treats concrete helper TestCase classes as warning tests

Aperta
#2,029 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
TypeScript
Stelle
17
Fork
4
Merge medio
59m
PR unite (30g)
131

Descrizione

## Problem

The generated plugin and core PHPUnit runners add every newly declared, non-abstract `PHPUnit\Framework\TestCase` subclass to the suite. Concrete helper/base classes that intentionally contain no runnable tests are therefore converted by PHPUnit 9 into warning tests:

```text
Warning
No tests found in class "A8C\CDN\Tests\TestCase".
```

This breaks valid suites under TeamCity's `--fail-on-warning` contract. It also diverges from PHPUnit configuration discovery, where helper classes may coexist with executable test classes in discovered files.

The affected code is in `packages/runtime-playground/src/phpunit-command-handlers.ts`, in both the plugin and WordPress core class-loading loops.

Exact WPCOM comparator evidence at `7ae9ae21ca30f7fc9724aecfb54a16f674f82a23` reproduced warnings for:

- `BkismetBaseTestCase`
- `A8C\CDN\Tests\TestCase`
- `A8C\GlotPressPlugins\Deliverables\DeliverablesTestCase`
- `GlotPressPluginsTestCase`
- `A8C\Test\Integration\WPCOM\Legacy\Upgrades\BaseAssignPayPalPaymentMethod`

WPCOM's hierarchy adds a second edge case: base classes expose an inherited static `suite()` factory. Checking only `hasMethod('suite')` still admits every concrete helper. A suite factory should qualify the candidate only when the candidate class declares it; normal inherited executable test methods must still qualify.

## Reproduction

Place a concrete helper beside a real test:

```php
class ConcreteTestHelper extends WP_UnitTestCase {
public function fixture(): string {
return 'fixture';
}
}

class ExampleTest extends WP_UnitTestCase {
public function test_example(): void {
$this->assertTrue(true);
}
}
```

Run the generated `wordpress.phpunit` recipe with warning failures enabled. The real test passes, but the helper creates `No tests found in class "ConcreteTestHelper"` and fails the suite.

## Expected

The generated runner should schedule only classes that PHPUnit recognizes as containing executable tests, plus candidate-owned static suite factories. Concrete helper classes should remain loadable without becoming warning tests.

## Acceptance criteria

- Concrete `TestCase` subclasses with no runnable tests are not added as suites.
- Abstract test bases remain skipped.
- `test*` methods, `@test` methods, and inherited runnable test methods remain discoverable.
- A static `suite()` declared by the candidate class remains supported.
- An inherited generic `suite()` does not qualify an otherwise empty helper class.
- Plugin and WordPress core generated runners share the same behavior.
- Unit coverage exercises both public and private PHPUnit `TestSuite` constructors supported by the runner.
- A real PHPUnit 9.6 Playground integration proves a helper and executable class can coexist without warnings and without mutating readonly source.

## Evidence

A local candidate on PR #1938's branch produced these exact-SHA results:

- `isolated:cdn`: failed before, passed after.
- `isolated:glotpress-plugins`: failed before, passed after.
- Nine-suite warning cluster: improved from 3/9 to 5/9, with all `No tests found in class` diagnostics removed and zero timeouts.

## Related

- #1937 owns clean-process bounded PHPUnit execution.
- PR #1938 implements that path and is where this defect surfaced.

## AI assistance

- **Model:** OpenAI GPT-5.6 Sol
- **Tool:** OpenCode
- **Used for:** Root-cause analysis, implementation and regression coverage, real PHPUnit integration verification, exact WPCOM comparator runs, and issue drafting under Chris Huber's direction.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia da packages/runtime-playground/src/phpunit-command-handlers.ts e confronta i cicli di caricamento delle classi del plugin e del core di WordPress. Esegui la recipe wordpress.phpunit generata con l’abilitazione degli errori per gli avvisi, quindi aggiungi la copertura dei test unitari per i costruttori pubblici e privati di PHPUnit TestSuite e un’integrazione Playground con PHPUnit 9.6. Il lavoro è completo quando le classi helper coesistono con test eseguibili senza avvisi e continuano a essere supportati sia i metodi eseguibili ereditati sia le factory di suite dichiarate dal candidato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
php, typescript
Ambito
testing-qa, tooling
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
62/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.