Automattic / Automattic/wp-codebox

wordpress.phpunit: database-type=mdi-native excludes 100% of candidates (excluded=1335, routed=0) on multisite + managed bootstrap

Offen
#2,500 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
TypeScript
Sterne
16
Forks
4
Ø Merge
59 Min.
Gemergte PRs (30 T.)
131

Beschreibung

## Summary

With `database-type=mdi-native`, `wordpress.phpunit` discovers the full candidate set and then **excludes 100% of it**, routing zero tests. The same component on `database-type=mysql` runs 1,177 tests normally.

```
FULL_SUITE_STANDALONE_PHP_SUMMARY:candidates=1335 selected=0 routed=0 excluded=1335 passed=0 failed=0
```

Discovery clearly works — it finds 1,335 candidates. Routing/selection is where everything is dropped.

## Reproduction

- wp-codebox: **`d4e0f713`** (current `main`), so this includes #2490 (mdi-native backend), #2494 (native multisite PHPUnit state), #2497 (explicit native MDI sources), and #2498 (retain PHPUnit failure artifacts).
- Consumer: Homeboy `wordpress` extension, managed install, provenance-valid (`homeboy extension setup wordpress` run clean; version reported `0.26.8` at the managed source path).
- Component under test: a WordPress plugin with
- `wp_codebox_multisite: true`
- `wp_codebox_phpunit_bootstrap_mode: "managed"`
- 101 test files / 1,335 PHPUnit candidates

Control vs variable, same component, same commit, only `database_type` changed:

| `database_type` | Result |
|---|---|
| `mysql` (external service) | **1,177 passed**, 7 skipped, 1,177 total, 577s |
| `mdi-native` | **0 passed, 0 failed, 0 skipped, 0 total** — `excluded=1335` |

## Behaviour detail

- The run does not fail fast at the point of exclusion. After reporting `excluded=1335` it continues until the harness budget is consumed. In one run it reported `test suite consumed 1336s of its 1500s budget (89%)` having executed nothing.
- With #2498 present the run surfaces `failed test command did not expose an observed failed test ID` rather than silently producing nothing, which is an improvement, but there is still no diagnostic explaining *why* 1,335 candidates were excluded.

## Why this matters

`mdi-native` is the difference between "tests need an external MySQL/MariaDB with provisioned credentials" and "tests need nothing." For consumers, that is the difference between a suite any contributor or agent can run locally and one gated behind secrets. Right now the backend validates, plans, and reports success at the config layer while running zero tests — so it cannot be trusted as a MySQL-compatible harness.

Credit where due: the consumer side handles this safely. Homeboy classifies zero executed tests as `invalid_evidence` → `passed: false`, so a release preflight on this backend is correctly blocked rather than shipping untested code. The danger is not a silent false pass in that toolchain, but the failure is silent enough that another consumer could plausibly treat "exit 0, no failures" as success.

## Asks

1. **Emit a reason for exclusion.** `excluded=1335` with no per-candidate or aggregate cause is not diagnosable. Even a single aggregate reason code would have made this self-service.
2. **Fix the routing/selection path for `mdi-native`** so candidates route the way they do under `mysql`. The identical candidate count (1,335) under both backends suggests discovery is shared and only selection/routing diverges.
3. **Fail fast on `selected=0` when `candidates>0`.** Consuming the full time budget after excluding everything wastes ~22 minutes per run and obscures the real error.

## Possibly relevant

The combination here is multisite + managed bootstrap. #2494 specifically addressed native multisite PHPUnit state, so this may be an unhandled remainder of that work rather than a general `mdi-native` failure — I have not tested single-site to isolate that, and would be happy to if it helps.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Beginne damit, die wordpress.phpunit-Komponente mit multisite und managed bootstrap beim wp-codebox-Commit d4e0f713 zu reproduzieren und database-type=mysql mit mdi-native zu vergleichen. Verfolge nach der Ermittlung den Auswahl- und Routing-Pfad der Kandidaten und prüfe anschließend die Ausschlussmeldungen sowie die Behandlung von selected=0. Die Arbeit ist abgeschlossen, wenn Kandidaten unter mdi-native geroutet werden, Ausschlüsse einen Grund offenlegen und ein Lauf mit null ausgewählten Elementen sofort fehlschlägt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
mariadb, mysql, php, typescript
Bereich
backend, databases, testing-qa
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Aktiv
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
52/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.