Automattic / Automattic/wp-codebox

PHPUnit runner treats concrete helper TestCase classes as warning tests

Open
#2,029 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16
Forks
4
Avg merge
59m
Merged PRs (30d)
131

Description

## Problem

The generated plugin and core PHPUnit runners add every newly declared, non-abstract `PHPUnit\Framework\TestCase` subclass to the suite. Concrete helper/base classes that intentionally contain no runnable tests are therefore converted by PHPUnit 9 into warning tests:

```text
Warning
No tests found in class "A8C\CDN\Tests\TestCase".
```

This breaks valid suites under TeamCity's `--fail-on-warning` contract. It also diverges from PHPUnit configuration discovery, where helper classes may coexist with executable test classes in discovered files.

The affected code is in `packages/runtime-playground/src/phpunit-command-handlers.ts`, in both the plugin and WordPress core class-loading loops.

Exact WPCOM comparator evidence at `7ae9ae21ca30f7fc9724aecfb54a16f674f82a23` reproduced warnings for:

- `BkismetBaseTestCase`
- `A8C\CDN\Tests\TestCase`
- `A8C\GlotPressPlugins\Deliverables\DeliverablesTestCase`
- `GlotPressPluginsTestCase`
- `A8C\Test\Integration\WPCOM\Legacy\Upgrades\BaseAssignPayPalPaymentMethod`

WPCOM's hierarchy adds a second edge case: base classes expose an inherited static `suite()` factory. Checking only `hasMethod('suite')` still admits every concrete helper. A suite factory should qualify the candidate only when the candidate class declares it; normal inherited executable test methods must still qualify.

## Reproduction

Place a concrete helper beside a real test:

```php
class ConcreteTestHelper extends WP_UnitTestCase {
public function fixture(): string {
return 'fixture';
}
}

class ExampleTest extends WP_UnitTestCase {
public function test_example(): void {
$this->assertTrue(true);
}
}
```

Run the generated `wordpress.phpunit` recipe with warning failures enabled. The real test passes, but the helper creates `No tests found in class "ConcreteTestHelper"` and fails the suite.

## Expected

The generated runner should schedule only classes that PHPUnit recognizes as containing executable tests, plus candidate-owned static suite factories. Concrete helper classes should remain loadable without becoming warning tests.

## Acceptance criteria

- Concrete `TestCase` subclasses with no runnable tests are not added as suites.
- Abstract test bases remain skipped.
- `test*` methods, `@test` methods, and inherited runnable test methods remain discoverable.
- A static `suite()` declared by the candidate class remains supported.
- An inherited generic `suite()` does not qualify an otherwise empty helper class.
- Plugin and WordPress core generated runners share the same behavior.
- Unit coverage exercises both public and private PHPUnit `TestSuite` constructors supported by the runner.
- A real PHPUnit 9.6 Playground integration proves a helper and executable class can coexist without warnings and without mutating readonly source.

## Evidence

A local candidate on PR #1938's branch produced these exact-SHA results:

- `isolated:cdn`: failed before, passed after.
- `isolated:glotpress-plugins`: failed before, passed after.
- Nine-suite warning cluster: improved from 3/9 to 5/9, with all `No tests found in class` diagnostics removed and zero timeouts.

## Related

- #1937 owns clean-process bounded PHPUnit execution.
- PR #1938 implements that path and is where this defect surfaced.

## AI assistance

- **Model:** OpenAI GPT-5.6 Sol
- **Tool:** OpenCode
- **Used for:** Root-cause analysis, implementation and regression coverage, real PHPUnit integration verification, exact WPCOM comparator runs, and issue drafting under Chris Huber's direction.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in packages/runtime-playground/src/phpunit-command-handlers.ts and compare the plugin and WordPress core class-loading loops. Run the generated wordpress.phpunit recipe with warning failures enabled, then add unit coverage for both public and private PHPUnit TestSuite constructors and a PHPUnit 9.6 Playground integration. Done means helper classes coexist with executable tests without warnings, while inherited runnable methods and candidate-declared suite factories remain supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, typescript
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.