Automattic / Automattic/wp-codebox
wordpress.phpunit: database-type=mdi-native excludes 100% of candidates (excluded=1335, routed=0) on multisite + managed bootstrap
- Vorherrschende Sprache
- TypeScript
- Sterne
- 16
- Forks
- 4
- Ø Merge
- 59 Min.
- Gemergte PRs (30 T.)
- 131
Beschreibung
## Summary
With `database-type=mdi-native`, `wordpress.phpunit` discovers the full candidate set and then **excludes 100% of it**, routing zero tests. The same component on `database-type=mysql` runs 1,177 tests normally.
```
FULL_SUITE_STANDALONE_PHP_SUMMARY:candidates=1335 selected=0 routed=0 excluded=1335 passed=0 failed=0
```
Discovery clearly works — it finds 1,335 candidates. Routing/selection is where everything is dropped.
## Reproduction
- wp-codebox: **`d4e0f713`** (current `main`), so this includes #2490 (mdi-native backend), #2494 (native multisite PHPUnit state), #2497 (explicit native MDI sources), and #2498 (retain PHPUnit failure artifacts).
- Consumer: Homeboy `wordpress` extension, managed install, provenance-valid (`homeboy extension setup wordpress` run clean; version reported `0.26.8` at the managed source path).
- Component under test: a WordPress plugin with
- `wp_codebox_multisite: true`
- `wp_codebox_phpunit_bootstrap_mode: "managed"`
- 101 test files / 1,335 PHPUnit candidates
Control vs variable, same component, same commit, only `database_type` changed:
| `database_type` | Result |
|---|---|
| `mysql` (external service) | **1,177 passed**, 7 skipped, 1,177 total, 577s |
| `mdi-native` | **0 passed, 0 failed, 0 skipped, 0 total** — `excluded=1335` |
## Behaviour detail
- The run does not fail fast at the point of exclusion. After reporting `excluded=1335` it continues until the harness budget is consumed. In one run it reported `test suite consumed 1336s of its 1500s budget (89%)` having executed nothing.
- With #2498 present the run surfaces `failed test command did not expose an observed failed test ID` rather than silently producing nothing, which is an improvement, but there is still no diagnostic explaining *why* 1,335 candidates were excluded.
## Why this matters
`mdi-native` is the difference between "tests need an external MySQL/MariaDB with provisioned credentials" and "tests need nothing." For consumers, that is the difference between a suite any contributor or agent can run locally and one gated behind secrets. Right now the backend validates, plans, and reports success at the config layer while running zero tests — so it cannot be trusted as a MySQL-compatible harness.
Credit where due: the consumer side handles this safely. Homeboy classifies zero executed tests as `invalid_evidence` → `passed: false`, so a release preflight on this backend is correctly blocked rather than shipping untested code. The danger is not a silent false pass in that toolchain, but the failure is silent enough that another consumer could plausibly treat "exit 0, no failures" as success.
## Asks
1. **Emit a reason for exclusion.** `excluded=1335` with no per-candidate or aggregate cause is not diagnosable. Even a single aggregate reason code would have made this self-service.
2. **Fix the routing/selection path for `mdi-native`** so candidates route the way they do under `mysql`. The identical candidate count (1,335) under both backends suggests discovery is shared and only selection/routing diverges.
3. **Fail fast on `selected=0` when `candidates>0`.** Consuming the full time budget after excluding everything wastes ~22 minutes per run and obscures the real error.
## Possibly relevant
The combination here is multisite + managed bootstrap. #2494 specifically addressed native multisite PHPUnit state, so this may be an unhandled remainder of that work rather than a general `mdi-native` failure — I have not tested single-site to isolate that, and would be happy to if it helps.
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Rechercherichtung
Beginne damit, die wordpress.phpunit-Komponente mit multisite und managed bootstrap beim wp-codebox-Commit d4e0f713 zu reproduzieren und database-type=mysql mit mdi-native zu vergleichen. Verfolge nach der Ermittlung den Auswahl- und Routing-Pfad der Kandidaten und prüfe anschließend die Ausschlussmeldungen sowie die Behandlung von selected=0. Die Arbeit ist abgeschlossen, wenn Kandidaten unter mdi-native geroutet werden, Ausschlüsse einen Grund offenlegen und ein Lauf mit null ausgewählten Elementen sofort fehlschlägt.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- mariadb, mysql, php, typescript
- Bereich
- backend, databases, testing-qa
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Aktiv
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 52/100