Automattic / Automattic/wp-codebox
wordpress.phpunit: database-type=mdi-native excludes 100% of candidates (excluded=1335, routed=0) on multisite + managed bootstrap
- Lingua principale
- TypeScript
- Stelle
- 16
- Fork
- 4
- Merge medio
- 59m
- PR unite (30g)
- 131
Descrizione
## Summary
With `database-type=mdi-native`, `wordpress.phpunit` discovers the full candidate set and then **excludes 100% of it**, routing zero tests. The same component on `database-type=mysql` runs 1,177 tests normally.
```
FULL_SUITE_STANDALONE_PHP_SUMMARY:candidates=1335 selected=0 routed=0 excluded=1335 passed=0 failed=0
```
Discovery clearly works — it finds 1,335 candidates. Routing/selection is where everything is dropped.
## Reproduction
- wp-codebox: **`d4e0f713`** (current `main`), so this includes #2490 (mdi-native backend), #2494 (native multisite PHPUnit state), #2497 (explicit native MDI sources), and #2498 (retain PHPUnit failure artifacts).
- Consumer: Homeboy `wordpress` extension, managed install, provenance-valid (`homeboy extension setup wordpress` run clean; version reported `0.26.8` at the managed source path).
- Component under test: a WordPress plugin with
- `wp_codebox_multisite: true`
- `wp_codebox_phpunit_bootstrap_mode: "managed"`
- 101 test files / 1,335 PHPUnit candidates
Control vs variable, same component, same commit, only `database_type` changed:
| `database_type` | Result |
|---|---|
| `mysql` (external service) | **1,177 passed**, 7 skipped, 1,177 total, 577s |
| `mdi-native` | **0 passed, 0 failed, 0 skipped, 0 total** — `excluded=1335` |
## Behaviour detail
- The run does not fail fast at the point of exclusion. After reporting `excluded=1335` it continues until the harness budget is consumed. In one run it reported `test suite consumed 1336s of its 1500s budget (89%)` having executed nothing.
- With #2498 present the run surfaces `failed test command did not expose an observed failed test ID` rather than silently producing nothing, which is an improvement, but there is still no diagnostic explaining *why* 1,335 candidates were excluded.
## Why this matters
`mdi-native` is the difference between "tests need an external MySQL/MariaDB with provisioned credentials" and "tests need nothing." For consumers, that is the difference between a suite any contributor or agent can run locally and one gated behind secrets. Right now the backend validates, plans, and reports success at the config layer while running zero tests — so it cannot be trusted as a MySQL-compatible harness.
Credit where due: the consumer side handles this safely. Homeboy classifies zero executed tests as `invalid_evidence` → `passed: false`, so a release preflight on this backend is correctly blocked rather than shipping untested code. The danger is not a silent false pass in that toolchain, but the failure is silent enough that another consumer could plausibly treat "exit 0, no failures" as success.
## Asks
1. **Emit a reason for exclusion.** `excluded=1335` with no per-candidate or aggregate cause is not diagnosable. Even a single aggregate reason code would have made this self-service.
2. **Fix the routing/selection path for `mdi-native`** so candidates route the way they do under `mysql`. The identical candidate count (1,335) under both backends suggests discovery is shared and only selection/routing diverges.
3. **Fail fast on `selected=0` when `candidates>0`.** Consuming the full time budget after excluding everything wastes ~22 minutes per run and obscures the real error.
## Possibly relevant
The combination here is multisite + managed bootstrap. #2494 specifically addressed native multisite PHPUnit state, so this may be an unhandled remainder of that work rather than a general `mdi-native` failure — I have not tested single-site to isolate that, and would be happy to if it helps.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Inizia riproducendo il componente wordpress.phpunit con multisite e managed bootstrap al commit d4e0f713 di wp-codebox, confrontando database-type=mysql con mdi-native. Traccia il percorso di selezione e routing dei candidati dopo la discovery, quindi esamina la segnalazione delle esclusioni e la gestione di selected=0. Il lavoro è completo quando i candidati vengono instradati sotto mdi-native, le esclusioni espongono un motivo e un'esecuzione con zero selezioni fallisce immediatamente.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- mariadb, mysql, php, typescript
- Ambito
- backend, databases, testing-qa
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Attiva
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 52/100