Automattic / Automattic/wp-codebox
Build a first-class adversarial WordPress fuzzing and fault-injection runtime
- Dominant language
- TypeScript
- Stars
- 16
- Forks
- 4
- Avg merge
- 59m
- Merged PRs (30d)
- 131
Description
## Problem
WP Codebox has excellent disposable WordPress runtimes, deterministic recipes, browser observation, multi-actor request gates, MySQL provisioning, checkpoints, replay packages, and artifact capture. Those primitives support authored fuzz campaigns, but they do not yet form a first-class adversarial fuzzing runtime capable of systematically finding unknown security, correctness, concurrency, and user-facing defects in arbitrary plugins, themes, or WordPress core.
Today a campaign author must supply most mutation logic, state exploration, service failure behavior, security oracles, corpus management, and minimization. Browser routing can allow/block/record traffic, but there is no transport-neutral fault model. `fuzzRun` records case/replay metadata, but it is not a coverage-guided mutation engine. MySQL is the only built-in disposable service, and its lifecycle cannot be disrupted mid-run.
The goal is not to promise that every zero-day will be found. The goal is to make WP Codebox capable of sustained, deterministic, hacker-grade adversarial testing that can aggressively attack a component's assumptions while preserving a safe, minimized, artifact-backed reproduction for every finding.
## Existing substrate to extend
- `packages/runtime-core/src/recipe-schema.ts`: `fuzzRun`, `distribution.serviceFakes`, overlays, services, artifacts, replay metadata.
- `packages/runtime-playground/src/browser-preview-routing.ts`: browser host allow/block/routing.
- `packages/runtime-playground/src/browser-multi-actor-scenario*.ts`: seeded actors, barriers, held requests, and deterministic schedules.
- `packages/runtime-playground/src/browser-capture-session.ts`: console, page errors, network, waterfall, DOM, screenshots, and diagnostics.
- `packages/cli/src/runtime-services.ts`: disposable MySQL/MariaDB service lifecycle.
- `packages/runtime-playground/src/runtime-snapshot.ts` and `replayable-wordpress-site-bundle.ts`: checkpoints and WordPress state replay.
- Typed artifact manifests, runtime attestations, mounted-file diffs, and sealed evidence bundles.
These should be composed, not replaced.
## Missing primitives
### 1. Stateful, coverage-guided mutation engine
Provide a generic campaign runner that can:
- maintain seed corpora and interesting-case queues;
- mutate scalar, structured, binary, multipart, JSON, XML, serialized PHP, block markup, shortcode, HTML, URL, header, cookie, and filesystem inputs;
- generate stateful action sequences rather than independent requests;
- use PHP/JavaScript/request/route/state coverage or novelty signals to retain cases;
- cross role, capability, multisite membership, authentication, locale, PHP, WordPress, and component-version matrices;
- enforce per-case resource budgets;
- deterministically replay a seed and normalized schedule;
- automatically shrink inputs and action sequences to a minimal reproduction.
WordPress-specific grammars belong in the WordPress extension. Core runtime mutation, scheduling, corpus, and minimization must remain generic.
### 2. Declarative transport fault model
Add typed request matching and scripted outcomes for browser and server-side traffic:
```json
{
"match": {
"host": "service.example",
"method": "POST",
"path": "/verify"
},
"sequence": [
{ "status": 500 },
{ "delayMs": 30000 },
{ "status": 200, "body": "malformed" }
]
}
```
Required fault classes:
- status/header/body substitution;
- malformed/truncated/chunked responses;
- arbitrary delay, jitter, bandwidth, and timeout stages;
- connection refusal, reset, half-close, and disconnect-after-N-bytes;
- DNS/host remapping;
- request/response corruption;
- deterministic response sequences and stateful counters;
- full request/fault/response evidence with secret redaction.
The same fault definition should work wherever technically possible across browser requests and WordPress HTTP transports, while reporting unsupported transport semantics explicitly.
### 3. Disruptable service environments
Extend `inputs.services` beyond MySQL and expose safe mid-run controls:
- Redis;
- SMTP/message sink;
- generic HTTP fixture service;
- optional object-storage/search fixtures where declared by an extension.
Fault controls should include stop, pause, restart, disconnect, latency, connection exhaustion, read-only mode, eviction/flush, malformed replies, disk-full/resource exhaustion, and recovery. MySQL needs mid-run disruption rather than provision-only lifecycle.
### 4. Server clock and scheduler control
Provide deterministic freeze, advance, skew, and restore controls for:
- PHP/WordPress time;
- browser time;
- cron/scheduled events;
- database comparisons where feasible.
This is required for expiry, nonce, token, cache, lock, retry, scheduling, and replay testing.
### 5. Adversarial browser and UI exploration
Build on browser scenarios to support:
- DOM-derived action/state graph exploration;
- generated click/fill/submit/navigation/back/reload sequences;
- rapid repeat, double-submit, stale-tab, multi-tab, and multi-user interactions;
- desktop/mobile/slow-device/locale/timezone/content-size matrices;
- malformed, empty, oversized, Unicode, bidi, and hostile rich-content inputs;
- layout oracles for overflow, overlap, clipping, invisible focus, dead controls, and stuck loading states;
- accessibility scanning;
- screenshot/DOM semantic regression comparison;
- automatic minimization to the shortest failing user journey.
Real browser engines and assistive technologies should be extension/provider capabilities, not assumptions in the generic runtime.
### 6. Security and integrity oracles
Campaigns need reusable, composable assertions for:
- authentication and authorization invariants;
- cross-user, cross-role, cross-site, and tenant isolation;
- nonce/CSRF and replay expectations;
- secret/PII leakage in responses, URLs, logs, HTML, screenshots, artifacts, and database state;
- unexpected privilege or session persistence;
- stored/reflected DOM execution indicators;
- filesystem escape and unexpected file mutation;
- database invariant and transactional consistency checks;
- duplicate side effects and idempotency;
- fail-open behavior after dependency/runtime faults;
- fatal, warning, notice, 5xx, unhandled rejection, console error, deadlock, livelock, timeout, and resource-growth detection.
The runtime should provide generic oracle plumbing; WordPress-specific policies should be registered by the WordPress extension or campaign.
### 7. Runtime instrumentation
Expose deterministic evidence useful for discovering novel paths:
- PHP and JavaScript coverage when available;
- WordPress hook, REST route, ability, AJAX action, cron, block render, CLI, and XML-RPC coverage via extension instrumentation;
- database query and mutation traces;
- filesystem mutation traces;
- cache operations;
- process memory, CPU, request duration, and leak trends;
- lock waits and concurrency schedules;
- sanitizers or engine diagnostics where supported by the backend.
Instrumentation must be opt-in, bounded, and artifact-safe.
### 8. Differential and compatibility fuzzing
Run the same minimized corpus across declared matrices:
- PHP versions;
- WordPress releases/trunk;
- component base versus candidate;
- SQLite versus MySQL/MariaDB;
- cache/service enabled versus disabled/faulted;
- browser/device profiles.
Classify findings as candidate regressions, pre-existing defects, platform differences, or nondeterminism.
### 9. First-class triage and replay
Every finding should produce a sealed reproduction containing:
- seed and corpus identity;
- minimized input and action schedule;
- exact component/runtime provenance;
- fault schedule;
- checkpoint or replayable WordPress state;
- relevant request, browser, database, filesystem, console, and runtime evidence;
- secret scan/redaction report;
- one-command replay instruction;
- stable finding fingerprint for deduplication.
A failed fuzz run without a replayable minimized case should be considered incomplete evidence.
## Safety requirements
- Network denied by default; external access must be declared and recorded.
- No host production credentials or state in a fuzz runtime.
- Bounded CPU, memory, disk, wall time, process count, and artifact size.
- Service and worker teardown must be guaranteed after crashes/timeouts.
- Secrets and personal data must be redacted before artifact publication.
- Dangerous payloads remain inside the disposable runtime and cannot address arbitrary host paths or services.
- Generic WP Codebox layers must not contain WordPress plugin, vendor, or transport-specific special cases; those belong in registered extension configuration.
## Proposed delivery phases
1. **Deterministic engine:** corpora, structured mutation, stateful sequences, novelty signals, shrinking, replay, and finding fingerprints.
2. **HTTP faults:** typed matchers, response sequences, latency/timeouts/resets, WordPress HTTP adapter, and browser adapter.
3. **Service faults:** Redis, SMTP, HTTP fixtures, and disruptable MySQL lifecycle.
4. **UI adversary:** DOM state exploration, layout/accessibility oracles, visual evidence, and journey minimization.
5. **Differential matrices:** component/runtime comparisons and regression classification.
6. **WordPress attack packs:** extension-owned grammars and oracles for REST, Abilities, AJAX, XML-RPC, blocks, options/meta, filesystem, cron, CLI, roles/capabilities, and multisite isolation.
## Acceptance criteria
- A recipe can run a deterministic adversarial campaign against an arbitrary mounted component without campaign-specific host scripting.
- A seeded intentionally vulnerable plugin/theme fixture proves discovery and minimization of authorization, injection, state corruption, race, external-service failure, and UI-state defects.
- At least one campaign demonstrates true concurrent workers plus deterministic schedule replay.
- At least one campaign demonstrates Redis, SMTP, HTTP, and MySQL fault/recovery behavior.
- At least one browser campaign automatically discovers and minimizes a user-facing interaction failure.
- Differential runs classify base/candidate and runtime-version behavior.
- Every retained finding has a stable fingerprint, bounded artifact bundle, secret scan, exact provenance, and one-command replay.
- Existing recipe, browser, service, artifact, and checkpoint contracts remain composable and backward compatible.
## Non-goals
- Claiming complete vulnerability detection or replacing expert security review.
- Baking one scanner, plugin, vendor, or WordPress-specific attack policy into the generic runtime.
- Allowing fuzz workloads to escape the Codebox sandbox or contact undeclared production services.
- Requiring every backend to support transport faults it cannot faithfully emulate; capability negotiation and explicit unsupported results are preferable to fake fidelity.
Contributor guide
No contributing guide indexed for this repository
Research direction
Begin by reading packages/runtime-core/src/recipe-schema.ts, packages/runtime-playground/src/browser-preview-routing.ts, browser-multi-actor-scenario*.ts, runtime-snapshot.ts, and packages/cli/src/runtime-services.ts. Start by scoping the deterministic engine phase and identifying the existing fuzzRun, replay, artifact, and service contracts it must compose with. Done requires a narrowly defined, tested phase with deterministic replay and backward-compatible contracts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, mysql, php, redis, typescript
- Domain
- backend, databases, devtools, security, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100