Automattic / Automattic/wp-codebox

wordpress.phpunit: database-type=mdi-native excludes 100% of candidates (excluded=1335, routed=0) on multisite + managed bootstrap

Open
#2,500 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16
Forks
4
Avg merge
59m
Merged PRs (30d)
131

Description

## Summary

With `database-type=mdi-native`, `wordpress.phpunit` discovers the full candidate set and then **excludes 100% of it**, routing zero tests. The same component on `database-type=mysql` runs 1,177 tests normally.

```
FULL_SUITE_STANDALONE_PHP_SUMMARY:candidates=1335 selected=0 routed=0 excluded=1335 passed=0 failed=0
```

Discovery clearly works — it finds 1,335 candidates. Routing/selection is where everything is dropped.

## Reproduction

- wp-codebox: **`d4e0f713`** (current `main`), so this includes #2490 (mdi-native backend), #2494 (native multisite PHPUnit state), #2497 (explicit native MDI sources), and #2498 (retain PHPUnit failure artifacts).
- Consumer: Homeboy `wordpress` extension, managed install, provenance-valid (`homeboy extension setup wordpress` run clean; version reported `0.26.8` at the managed source path).
- Component under test: a WordPress plugin with
- `wp_codebox_multisite: true`
- `wp_codebox_phpunit_bootstrap_mode: "managed"`
- 101 test files / 1,335 PHPUnit candidates

Control vs variable, same component, same commit, only `database_type` changed:

| `database_type` | Result |
|---|---|
| `mysql` (external service) | **1,177 passed**, 7 skipped, 1,177 total, 577s |
| `mdi-native` | **0 passed, 0 failed, 0 skipped, 0 total** — `excluded=1335` |

## Behaviour detail

- The run does not fail fast at the point of exclusion. After reporting `excluded=1335` it continues until the harness budget is consumed. In one run it reported `test suite consumed 1336s of its 1500s budget (89%)` having executed nothing.
- With #2498 present the run surfaces `failed test command did not expose an observed failed test ID` rather than silently producing nothing, which is an improvement, but there is still no diagnostic explaining *why* 1,335 candidates were excluded.

## Why this matters

`mdi-native` is the difference between "tests need an external MySQL/MariaDB with provisioned credentials" and "tests need nothing." For consumers, that is the difference between a suite any contributor or agent can run locally and one gated behind secrets. Right now the backend validates, plans, and reports success at the config layer while running zero tests — so it cannot be trusted as a MySQL-compatible harness.

Credit where due: the consumer side handles this safely. Homeboy classifies zero executed tests as `invalid_evidence` → `passed: false`, so a release preflight on this backend is correctly blocked rather than shipping untested code. The danger is not a silent false pass in that toolchain, but the failure is silent enough that another consumer could plausibly treat "exit 0, no failures" as success.

## Asks

1. **Emit a reason for exclusion.** `excluded=1335` with no per-candidate or aggregate cause is not diagnosable. Even a single aggregate reason code would have made this self-service.
2. **Fix the routing/selection path for `mdi-native`** so candidates route the way they do under `mysql`. The identical candidate count (1,335) under both backends suggests discovery is shared and only selection/routing diverges.
3. **Fail fast on `selected=0` when `candidates>0`.** Consuming the full time budget after excluding everything wastes ~22 minutes per run and obscures the real error.

## Possibly relevant

The combination here is multisite + managed bootstrap. #2494 specifically addressed native multisite PHPUnit state, so this may be an unhandled remainder of that work rather than a general `mdi-native` failure — I have not tested single-site to isolate that, and would be happy to if it helps.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the wordpress.phpunit component with multisite and managed bootstrap at wp-codebox commit d4e0f713, comparing database-type=mysql with mdi-native. Trace the candidate selection and routing path after discovery, then inspect the exclusion reporting and selected=0 handling. Done means candidates route under mdi-native, exclusions expose a reason, and a zero-selection run fails fast.

Written by the indexing model from the issue text.

Assessment

Tech stack
mariadb, mysql, php, typescript
Domain
backend, databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.